2024-04-16T11:16:01.758 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T11:16:01.758 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi --verbose --archive /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071 --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/orig.config.yaml 2024-04-16T11:16:01.809 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T11:16:01.896 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071 branch: wip-vshankar-testing-20240412.042930 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: vshankar@redhat.com first_in_suite: false job_id: '7658071' kernel: flavor: default kdb: true sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/wip-64502-quincy-3 ' name: vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240412.042930 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-64502-quincy-3 sha1: 96e30c031480d623ee32b6336cfd0a901a764a51 owner: scheduled_vshankar@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9621 sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: wip-64502-quincy-3 suite_path: /home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa suite_relpath: qa suite_repo: https://github.com/vshankar/ceph.git suite_sha1: 96e30c031480d623ee32b6336cfd0a901a764a51 targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6Bk5baccvIJJs+tGtUEIPlBdRsUeJyJlsHTonddFEJdBksjvfLxzn/x1+9a4XK5hmD63IZlRVFjCAPB+kVKNw= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYlwGYIuSCqXHAUcISzjQxeGmJRcvFIiDt2jY7gf2HM4FuIi03ghhWtNQbELudwXDb7dYy73Vzte/GTo/A6X3I= tasks: - install: branch: wip-64502-quincy-3 exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/vshankar/src/github.com_vshankar_ceph_wip-64502-quincy-3/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_11:10:55 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T11:16:01.896 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa; will attempt to use it 2024-04-16T11:16:01.920 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks 2024-04-16T11:16:01.921 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T11:16:01.922 INFO:teuthology.task.internal:Checking packages... 2024-04-16T11:16:01.943 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '218a691417ef57c5d22a8d5cf3c5f41f7b542838' 2024-04-16T11:16:01.943 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:16:01.943 INFO:teuthology.packaging:ref: None 2024-04-16T11:16:01.943 INFO:teuthology.packaging:tag: None 2024-04-16T11:16:01.943 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240412.042930 2024-04-16T11:16:01.943 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:16:01.943 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240412.042930 2024-04-16T11:16:02.086 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2884.g218a6914 2024-04-16T11:16:02.087 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T11:16:02.094 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T11:16:02.094 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T11:16:02.115 INFO:teuthology.task.internal:Saving configuration 2024-04-16T11:16:02.132 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T11:16:02.139 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T11:16:02.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 11:11:05.089780', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6Bk5baccvIJJs+tGtUEIPlBdRsUeJyJlsHTonddFEJdBksjvfLxzn/x1+9a4XK5hmD63IZlRVFjCAPB+kVKNw='} 2024-04-16T11:16:02.184 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 11:11:05.087295', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYlwGYIuSCqXHAUcISzjQxeGmJRcvFIiDt2jY7gf2HM4FuIi03ghhWtNQbELudwXDb7dYy73Vzte/GTo/A6X3I='} 2024-04-16T11:16:02.184 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T11:16:02.191 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-16T11:16:02.191 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-16T11:16:02.191 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T11:16:02.251 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f67b24600d0>, signals=[15]) 2024-04-16T11:16:02.251 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T11:16:02.258 INFO:teuthology.task.internal:Opening connections... 2024-04-16T11:16:02.259 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:16:02.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:16:02.366 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T11:16:02.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:16:02.465 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T11:16:02.472 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-04-16T11:16:02.489 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-04-16T11:16:02.489 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-04-16T11:16:02.545 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T11:16:02.546 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T11:16:02.547 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T11:16:02.548 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-04-16T11:16:02.568 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-04-16T11:16:02.585 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-04-16T11:16:02.585 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T11:16:02.642 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-04-16T11:16:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-04-16T11:16:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T11:16:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-04-16T11:16:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T11:16:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T11:16:02.643 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T11:16:02.643 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-16T11:16:02.659 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T11:16:02.669 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T11:16:02.676 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T11:16:02.676 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-04-16T11:16:02.678 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-04-16T11:16:02.701 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T11:16:02.709 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T11:16:02.709 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-04-16T11:16:02.734 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-04-16T11:16:02.778 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T11:16:02.832 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T11:16:02.847 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'} 2024-04-16T11:16:02.847 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}}, timeout 300 2024-04-16T11:16:02.847 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T11:16:02.847 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T11:16:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T11:16:02.863 DEBUG:teuthology.task.kernel:sha1 for host.a is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:02.863 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T11:16:02.863 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-04-16T11:16:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T11:16:02.864 DEBUG:teuthology.task.kernel:sha1 for host.b is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:02.864 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T11:16:02.865 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-16T11:16:02.919 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-361.el9.x86_64 2024-04-16T11:16:02.920 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:02.920 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T11:16:02.920 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:02.923 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-04-16T11:16:02.924 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:02.924 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T11:16:02.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:03.143 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum upgrade -y linux-firmware 2024-04-16T11:16:03.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum upgrade -y linux-firmware 2024-04-16T11:16:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-04-16T11:16:05.521 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 8.5 MB/s | 6.7 MB 00:00 2024-04-16T11:16:07.026 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-04-16T11:16:07.054 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-04-16T11:16:09.065 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-04-16T11:16:10.303 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-04-16T11:16:11.925 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-16T11:16:12.184 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-16T11:16:16.610 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.2 MB 00:00 2024-04-16T11:16:18.295 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-16T11:16:19.294 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.2 MB 00:03 2024-04-16T11:16:20.245 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T11:16:20.709 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-04-16T11:16:22.504 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-16T11:16:25.529 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-04-16T11:16:27.735 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-16T11:16:28.088 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:16:28.099 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:16:28.100 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:16:28.100 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 1 Package 2024-04-16T11:16:28.100 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:16:28.100 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 375 M 2024-04-16T11:16:28.100 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:16:30.337 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:16:30.345 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:16:30.345 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:16:30.345 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 1 Package 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 375 M 2024-04-16T11:16:30.346 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:16:39.753 INFO:teuthology.orchestra.run.smithi044.stdout:linux-firmware-20240219-143.el9.noarch.rpm 40 MB/s | 375 MB 00:09 2024-04-16T11:16:39.754 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:16:39.754 INFO:teuthology.orchestra.run.smithi044.stdout:Total 40 MB/s | 375 MB 00:09 2024-04-16T11:16:39.755 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:16:39.848 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:16:39.848 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:16:40.962 INFO:teuthology.orchestra.run.smithi045.stdout:linux-firmware-20240219-143.el9.noarch.rpm 29 MB/s | 375 MB 00:12 2024-04-16T11:16:40.963 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:16:40.963 INFO:teuthology.orchestra.run.smithi045.stdout:Total 29 MB/s | 375 MB 00:12 2024-04-16T11:16:40.963 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:16:41.076 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:16:41.076 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:16:42.340 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:16:42.340 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:16:43.571 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:16:43.572 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:16:48.853 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:16:49.102 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T11:16:49.205 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T11:16:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:16:50.425 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T11:16:50.514 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T11:16:50.560 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T11:16:50.560 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T11:16:50.976 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T11:16:50.976 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:16:50.976 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-04-16T11:16:50.976 INFO:teuthology.orchestra.run.smithi044.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T11:16:50.976 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:16:50.976 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:16:51.236 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.a... 2024-04-16T11:16:51.237 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:51.381 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T11:16:51.381 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T11:16:51.384 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi044.front.sepia.ceph.com to complete... 2024-04-16T11:16:51.805 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T11:16:51.806 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T11:16:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T11:16:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:16:52.223 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-04-16T11:16:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T11:16:52.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:16:52.223 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:16:52.508 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.b... 2024-04-16T11:16:52.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T11:16:52.735 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T11:16:52.735 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T11:16:52.737 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi045.front.sepia.ceph.com to complete... 2024-04-16T11:17:14.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16 11:17:14 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T11:17:14.333 INFO:teuthology.orchestra.run.smithi045.stderr:FINISHED --2024-04-16 11:17:14-- 2024-04-16T11:17:14.333 INFO:teuthology.orchestra.run.smithi045.stderr:Total wall clock time: 22s 2024-04-16T11:17:14.333 INFO:teuthology.orchestra.run.smithi045.stderr:Downloaded: 1 files, 341M in 21s (16.1 MB/s) 2024-04-16T11:17:14.336 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.b... 2024-04-16T11:17:14.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T11:17:15.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16 11:17:15 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T11:17:15.200 INFO:teuthology.orchestra.run.smithi044.stderr:FINISHED --2024-04-16 11:17:15-- 2024-04-16T11:17:15.200 INFO:teuthology.orchestra.run.smithi044.stderr:Total wall clock time: 24s 2024-04-16T11:17:15.200 INFO:teuthology.orchestra.run.smithi044.stderr:Downloaded: 1 files, 341M in 24s (14.5 MB/s) 2024-04-16T11:17:15.204 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.a... 2024-04-16T11:17:15.204 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T11:17:16.493 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying... ######################################## 2024-04-16T11:17:16.547 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing... ######################################## 2024-04-16T11:17:16.631 INFO:teuthology.orchestra.run.smithi045.stdout:Updating / installing... 2024-04-16T11:17:17.362 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying... ######################################## 2024-04-16T11:17:17.422 INFO:teuthology.orchestra.run.smithi044.stdout:Preparing... ######################################## 2024-04-16T11:17:17.506 INFO:teuthology.orchestra.run.smithi044.stdout:Updating / installing... 2024-04-16T11:17:24.931 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T11:17:25.795 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T11:17:41.701 INFO:teuthology.orchestra.run.smithi045.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T11:17:42.865 INFO:teuthology.orchestra.run.smithi044.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T11:18:05.340 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T11:18:05.341 DEBUG:teuthology.orchestra.run.smithi045:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T11:18:05.410 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.410 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.410 INFO:teuthology.orchestra.run.smithi045.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.410 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T11:18:05.411 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T11:18:05.412 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T11:18:05.413 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T11:18:05.414 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T11:18:05.415 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T11:18:05.416 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T11:18:05.417 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T11:18:05.418 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T11:18:05.419 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T11:18:05.420 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T11:18:05.421 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T11:18:05.422 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T11:18:05.423 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T11:18:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T11:18:05.425 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T11:18:05.426 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T11:18:05.427 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T11:18:05.428 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T11:18:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T11:18:05.430 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T11:18:05.431 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T11:18:05.432 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T11:18:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T11:18:05.434 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T11:18:05.435 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T11:18:05.436 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T11:18:05.437 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T11:18:05.438 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T11:18:05.439 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T11:18:05.440 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T11:18:05.441 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T11:18:05.442 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T11:18:05.443 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T11:18:05.444 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T11:18:05.445 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T11:18:05.446 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T11:18:05.447 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T11:18:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T11:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T11:18:05.450 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T11:18:05.451 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T11:18:05.452 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T11:18:05.453 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T11:18:05.454 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T11:18:05.455 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T11:18:05.456 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T11:18:05.457 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T11:18:05.458 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T11:18:05.459 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T11:18:05.460 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T11:18:05.461 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T11:18:05.462 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T11:18:05.463 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T11:18:05.464 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T11:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T11:18:05.466 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T11:18:05.467 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T11:18:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T11:18:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T11:18:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T11:18:05.471 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T11:18:05.472 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T11:18:05.473 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T11:18:05.474 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T11:18:05.475 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T11:18:05.476 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T11:18:05.477 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T11:18:05.478 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T11:18:05.479 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T11:18:05.480 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T11:18:05.481 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T11:18:05.482 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T11:18:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T11:18:05.484 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T11:18:05.485 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T11:18:05.486 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T11:18:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T11:18:05.488 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T11:18:05.489 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T11:18:05.490 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T11:18:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T11:18:05.492 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T11:18:05.493 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T11:18:05.494 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T11:18:05.495 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T11:18:05.496 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T11:18:05.497 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T11:18:05.498 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T11:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T11:18:05.500 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T11:18:05.501 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T11:18:05.502 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T11:18:05.503 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T11:18:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T11:18:05.506 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T11:18:05.507 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T11:18:05.508 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T11:18:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T11:18:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T11:18:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T11:18:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T11:18:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T11:18:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T11:18:05.515 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T11:18:05.516 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T11:18:05.517 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T11:18:05.518 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T11:18:05.519 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T11:18:05.520 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T11:18:05.521 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T11:18:05.522 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T11:18:05.523 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T11:18:05.524 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T11:18:05.525 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T11:18:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T11:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T11:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T11:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T11:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T11:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T11:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T11:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T11:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T11:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T11:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T11:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T11:18:05.541 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.541 DEBUG:teuthology.orchestra.run.smithi045:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T11:18:05.563 INFO:teuthology.orchestra.run.smithi045.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T11:18:05.563 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T11:18:05.563 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:for file in vmlinuz System.map config; do 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: fi 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T11:18:05.564 INFO:teuthology.orchestra.run.smithi045.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T11:18:05.565 INFO:teuthology.orchestra.run.smithi045.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T11:18:05.565 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.565 INFO:teuthology.orchestra.run.smithi045.stdout:fi 2024-04-16T11:18:05.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-04-16T11:18:05.654 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-04-16T11:18:05.654 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-04-16T11:18:05.654 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-04-16T11:18:05.654 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-04-16T11:18:05.655 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:18:05.656 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T11:18:05.656 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T11:18:05.656 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T11:18:05.656 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-04-16T11:18:05.656 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:18:05.656 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T11:18:05.657 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T11:18:05.657 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:05.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T11:18:06.276 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-04-16T11:18:06.521 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T11:18:06.521 DEBUG:teuthology.orchestra.run.smithi044:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T11:18:06.597 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.597 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.597 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.597 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T11:18:06.597 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T11:18:06.597 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T11:18:06.598 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T11:18:06.599 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T11:18:06.600 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T11:18:06.601 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T11:18:06.602 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T11:18:06.603 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T11:18:06.604 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T11:18:06.605 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T11:18:06.606 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T11:18:06.607 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T11:18:06.608 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T11:18:06.609 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T11:18:06.610 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T11:18:06.611 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T11:18:06.612 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T11:18:06.613 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T11:18:06.614 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T11:18:06.615 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T11:18:06.616 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T11:18:06.617 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T11:18:06.618 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T11:18:06.619 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T11:18:06.620 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T11:18:06.621 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T11:18:06.622 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T11:18:06.623 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T11:18:06.624 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T11:18:06.625 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T11:18:06.626 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T11:18:06.627 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T11:18:06.628 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T11:18:06.629 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T11:18:06.630 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T11:18:06.631 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T11:18:06.632 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T11:18:06.633 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T11:18:06.634 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T11:18:06.635 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T11:18:06.636 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T11:18:06.637 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T11:18:06.638 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T11:18:06.639 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T11:18:06.640 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T11:18:06.641 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T11:18:06.642 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T11:18:06.643 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T11:18:06.644 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T11:18:06.645 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T11:18:06.646 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T11:18:06.647 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T11:18:06.648 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T11:18:06.649 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T11:18:06.650 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T11:18:06.651 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T11:18:06.652 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T11:18:06.653 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T11:18:06.654 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T11:18:06.655 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T11:18:06.656 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T11:18:06.657 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T11:18:06.658 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T11:18:06.659 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T11:18:06.660 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T11:18:06.661 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T11:18:06.662 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T11:18:06.663 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T11:18:06.664 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T11:18:06.665 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T11:18:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T11:18:06.667 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T11:18:06.668 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T11:18:06.669 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T11:18:06.670 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T11:18:06.671 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T11:18:06.672 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T11:18:06.673 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T11:18:06.674 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T11:18:06.675 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T11:18:06.676 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T11:18:06.677 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T11:18:06.678 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T11:18:06.679 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T11:18:06.680 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T11:18:06.681 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T11:18:06.682 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T11:18:06.683 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T11:18:06.684 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T11:18:06.685 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T11:18:06.686 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T11:18:06.687 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T11:18:06.688 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T11:18:06.689 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T11:18:06.690 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T11:18:06.691 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T11:18:06.692 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T11:18:06.693 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T11:18:06.694 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T11:18:06.695 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T11:18:06.696 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T11:18:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T11:18:06.698 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T11:18:06.699 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T11:18:06.700 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T11:18:06.701 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T11:18:06.702 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T11:18:06.703 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T11:18:06.704 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T11:18:06.705 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T11:18:06.706 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T11:18:06.707 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T11:18:06.708 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T11:18:06.709 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T11:18:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T11:18:06.711 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T11:18:06.712 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T11:18:06.713 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T11:18:06.714 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T11:18:06.715 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T11:18:06.716 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T11:18:06.717 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T11:18:06.720 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.720 DEBUG:teuthology.orchestra.run.smithi044:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T11:18:06.744 INFO:teuthology.orchestra.run.smithi044.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T11:18:06.744 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T11:18:06.744 INFO:teuthology.orchestra.run.smithi044.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:for file in vmlinuz System.map config; do 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: fi 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:done 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-04-16T11:18:06.745 INFO:teuthology.orchestra.run.smithi044.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T11:18:06.746 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T11:18:06.746 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.746 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-04-16T11:18:06.746 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-04-16T11:18:06.835 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-04-16T11:18:06.835 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-04-16T11:18:06.835 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-04-16T11:18:06.835 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-04-16T11:18:06.835 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-04-16T11:18:06.835 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T11:18:06.836 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T11:18:06.837 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T11:18:06.837 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-04-16T11:18:06.837 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:18:06.837 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T11:18:06.837 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T11:18:06.838 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:06.838 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T11:18:07.424 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-04-16T11:18:07.652 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T11:18:07.691 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-04-16T11:18:07.694 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-16T11:18:07.709 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.1leHDJ5wnt 2024-04-16T11:18:07.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1leHDJ5wnt 2024-04-16T11:18:07.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.1leHDJ5wnt 2024-04-16T11:18:07.956 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.1leHDJ5wnt is 6KB 2024-04-16T11:18:08.005 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.1leHDJ5wnt 2024-04-16T11:18:08.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T11:18:08.088 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T11:18:08.088 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T11:18:08.088 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T11:18:08.088 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T11:18:08.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:08.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-04-16T11:18:08.982 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T11:18:09.034 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-04-16T11:18:09.036 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-04-16T11:18:09.051 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.WOSJI97jCr 2024-04-16T11:18:09.051 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WOSJI97jCr 2024-04-16T11:18:09.122 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.WOSJI97jCr 2024-04-16T11:18:09.387 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.WOSJI97jCr is 6KB 2024-04-16T11:18:09.407 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.WOSJI97jCr 2024-04-16T11:18:09.421 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T11:18:09.489 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T11:18:09.489 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T11:18:09.489 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T11:18:09.490 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T11:18:09.491 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:18:09.661 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-04-16T11:18:38.261 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-16T11:18:38.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T11:18:38.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:18:39.666 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-16T11:18:39.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-16T11:18:39.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:19:38.314 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T11:19:39.671 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T11:19:47.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T11:19:47.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:19:48.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-16T11:19:48.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:19:50.388 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-16T11:19:51.199 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T11:19:51.199 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T11:19:51.199 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-16T11:19:51.240 INFO:teuthology.orchestra.run.smithi045.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:19:51.240 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:19:51.240 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T11:19:51.240 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T11:19:52.241 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-16T11:19:52.241 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T11:19:52.371 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-04-16T11:19:52.423 DEBUG:teuthology.parallel:result is None 2024-04-16T11:19:56.405 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-04-16T11:19:57.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-16T11:19:57.125 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T11:19:57.125 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-04-16T11:19:57.169 INFO:teuthology.orchestra.run.smithi044.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:19:57.169 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T11:19:57.169 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T11:19:57.169 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T11:19:58.170 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-16T11:19:58.170 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T11:19:58.290 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2024-04-16T11:19:58.355 DEBUG:teuthology.parallel:result is None 2024-04-16T11:19:58.355 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T11:19:58.362 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T11:19:58.363 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T11:19:58.368 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T11:19:58.421 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T11:19:58.432 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T11:19:58.445 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T11:19:58.445 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T11:19:58.454 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T11:19:58.538 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T11:19:58.547 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T11:19:58.547 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T11:19:58.552 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T11:19:58.633 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T11:19:58.662 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T11:19:58.674 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T11:19:58.699 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T11:19:58.703 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T11:19:58.711 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T11:19:58.711 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T11:19:58.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T11:19:58.791 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T11:19:58.803 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T11:19:58.803 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T11:19:58.807 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T11:19:58.846 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T11:19:58.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T11:19:59.056 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T11:19:59.126 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T11:19:59.217 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:19:59.217 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T11:19:59.294 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T11:19:59.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T11:19:59.451 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T11:19:59.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T11:19:59.596 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:19:59.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T11:19:59.683 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-04-16T11:19:59.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-04-16T11:19:59.793 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T11:19:59.801 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T11:20:00.308 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T11:20:00.317 INFO:teuthology.task.internal:Starting timer... 2024-04-16T11:20:00.317 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T11:20:00.334 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T11:20:00.347 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-16T11:20:00.347 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-04-16T11:20:00.490 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-04-16T11:20:00.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-04-16T11:20:00.595 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-04-16T11:20:00.599 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T11:20:00.600 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-04-16T11:20:00.648 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-04-16T11:20:00.649 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-04-16T11:20:00.698 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-04-16T11:20:00.698 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi045.front.sepia.ceph.com': 'permissive'} 2024-04-16T11:20:00.698 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T11:20:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T11:20:00.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T11:20:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T11:20:00.862 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T11:20:00.862 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-04-16T11:20:00.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-04-16T11:20:01.035 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T11:20:01.049 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T11:20:01.056 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T11:20:01.204 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T11:20:01.219 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T11:20:01.221 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T11:26:11.708 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com')] 2024-04-16T11:26:11.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-16T11:26:11.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:26:11.863 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-04-16T11:26:12.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-16T11:26:12.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T11:26:12.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T11:26:12.178 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-16T11:26:12.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-16T11:26:12.334 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T11:26:12.346 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T11:26:12.347 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T11:26:12.347 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T11:26:12.355 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T11:26:12.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T11:26:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T11:26:12.461 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T11:26:12.515 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T11:26:12.519 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T11:26:12.609 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-04-16T11:26:12.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-04-16T11:26:12.652 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-04-16T11:26:12.658 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-04-16T11:26:12.706 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T11:26:12.709 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T11:26:12.766 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T11:26:12.768 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T11:26:12.910 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-04-16T11:26:12.913 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-04-16T11:26:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T11:26:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-04-16T11:26:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T11:26:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-04-16T11:26:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T11:26:12.925 INFO:teuthology.run_tasks:Running task install... 2024-04-16T11:26:12.934 DEBUG:teuthology.task.install:project ceph 2024-04-16T11:26:12.934 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'}} 2024-04-16T11:26:12.934 DEBUG:teuthology.task.install:config {'branch': 'wip-64502-quincy-3', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-16T11:26:12.934 INFO:teuthology.task.install:Using flavor: default 2024-04-16T11:26:12.985 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T11:26:12.985 INFO:teuthology.task.install:extra packages: [] 2024-04-16T11:26:12.985 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'wip-64502-quincy-3', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-16T11:26:12.985 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:26:12.985 INFO:teuthology.packaging:ref: None 2024-04-16T11:26:12.986 INFO:teuthology.packaging:tag: None 2024-04-16T11:26:12.986 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T11:26:12.986 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:26:12.986 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T11:26:12.988 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'wip-64502-quincy-3', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-16T11:26:12.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:26:12.988 INFO:teuthology.packaging:ref: None 2024-04-16T11:26:12.988 INFO:teuthology.packaging:tag: None 2024-04-16T11:26:12.988 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T11:26:12.988 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:26:12.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T11:26:13.140 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/ 2024-04-16T11:26:13.141 INFO:teuthology.task.install.rpm:Package version is 17.2.7-847.g45317f95 2024-04-16T11:26:13.236 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/ 2024-04-16T11:26:13.236 INFO:teuthology.task.install.rpm:Package version is 17.2.7-847.g45317f95 2024-04-16T11:26:13.251 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T11:26:13.251 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:26:13.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T11:26:13.341 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T11:26:13.341 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:26:13.341 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T11:26:13.356 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T11:26:13.357 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:26:13.357 INFO:teuthology.packaging:ref: None 2024-04-16T11:26:13.357 INFO:teuthology.packaging:tag: None 2024-04-16T11:26:13.357 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T11:26:13.357 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:26:13.357 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/wip-64502-quincy-3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T11:26:13.449 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T11:26:13.449 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:26:13.449 INFO:teuthology.packaging:ref: None 2024-04-16T11:26:13.449 INFO:teuthology.packaging:tag: None 2024-04-16T11:26:13.449 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T11:26:13.450 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:26:13.450 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/wip-64502-quincy-3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T11:26:13.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T11:26:13.596 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T11:26:13.641 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T11:26:13.729 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T11:26:13.734 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-04-16T11:26:13.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-04-16T11:26:13.816 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-04-16T11:26:13.821 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-04-16T11:26:14.114 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-04-16T11:26:14.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-04-16T11:26:14.198 INFO:teuthology.orchestra.run.smithi044.stdout:74 files removed 2024-04-16T11:26:14.239 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw 2024-04-16T11:26:15.201 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 148 kB/s | 77 kB 00:00 2024-04-16T11:26:15.443 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 113 kB/s | 77 kB 00:00 2024-04-16T11:26:15.892 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 18 kB/s | 11 kB 00:00 2024-04-16T11:26:16.252 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 15 kB/s | 11 kB 00:00 2024-04-16T11:26:16.700 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 2.5 kB/s | 1.8 kB 00:00 2024-04-16T11:26:16.702 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 4.9 kB/s | 1.8 kB 00:00 2024-04-16T11:26:17.799 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.7 MB 00:01 2024-04-16T11:26:17.837 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 6.3 MB/s | 6.7 MB 00:01 2024-04-16T11:26:19.681 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-04-16T11:26:19.713 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-04-16T11:26:21.492 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-04-16T11:26:23.072 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-04-16T11:26:23.485 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-16T11:26:25.371 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-16T11:26:28.958 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.2 MB 00:01 2024-04-16T11:26:30.468 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.2 MB 00:01 2024-04-16T11:26:31.020 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-16T11:26:32.131 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-04-16T11:26:34.038 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 21 MB 00:02 2024-04-16T11:26:34.988 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-04-16T11:26:40.229 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-16T11:26:41.090 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-16T11:26:43.380 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 14 M 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.1 M 2024-04-16T11:26:43.389 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:17.2.7-847.g45317f95.el9 ceph 5.0 M 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 22 M 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 639 k 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T11:26:43.390 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 457 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 4.9 M 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:17.2.7-847.g45317f95.el9 ceph 45 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 117 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:17.2.7-847.g45317f95.el9 ceph 156 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:17.2.7-847.g45317f95.el9 ceph 320 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 293 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 100 k 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:26:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:26:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:26:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Install 22 Packages 2024-04-16T11:26:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-04-16T11:26:43.392 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:26:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 58 M 2024-04-16T11:26:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:26:44.269 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:26:44.281 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 14 M 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.1 M 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:17.2.7-847.g45317f95.el9 ceph 5.0 M 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 22 M 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T11:26:44.282 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 639 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 457 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 4.9 M 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse x86_64 2:17.2.7-847.g45317f95.el9 ceph 45 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 117 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:17.2.7-847.g45317f95.el9 ceph 156 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:17.2.7-847.g45317f95.el9 ceph 320 k 2024-04-16T11:26:44.283 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 293 k 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 100 k 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout:Install 22 Packages 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 2 Packages 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:26:44.284 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 58 M 2024-04-16T11:26:44.285 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:26:45.006 INFO:teuthology.orchestra.run.smithi045.stdout:(1/24): ceph-common-17.2.7-847.g45317f95.el9.x8 15 MB/s | 22 MB 00:01 2024-04-16T11:26:45.089 INFO:teuthology.orchestra.run.smithi045.stdout:(2/24): ceph-selinux-17.2.7-847.g45317f95.el9.x 307 kB/s | 25 kB 00:00 2024-04-16T11:26:45.189 INFO:teuthology.orchestra.run.smithi045.stdout:(3/24): libcephfs2-17.2.7-847.g45317f95.el9.x86 6.3 MB/s | 639 kB 00:00 2024-04-16T11:26:45.281 INFO:teuthology.orchestra.run.smithi045.stdout:(4/24): libradosstriper1-17.2.7-847.g45317f95.e 4.9 MB/s | 457 kB 00:00 2024-04-16T11:26:45.856 INFO:teuthology.orchestra.run.smithi045.stdout:(5/24): ceph-base-17.2.7-847.g45317f95.el9.x86_ 2.1 MB/s | 5.0 MB 00:02 2024-04-16T11:26:45.981 INFO:teuthology.orchestra.run.smithi045.stdout:(6/24): librgw2-17.2.7-847.g45317f95.el9.x86_64 7.0 MB/s | 4.9 MB 00:00 2024-04-16T11:26:46.007 INFO:teuthology.orchestra.run.smithi045.stdout:(7/24): python3-ceph-argparse-17.2.7-847.g45317 303 kB/s | 45 kB 00:00 2024-04-16T11:26:46.049 INFO:teuthology.orchestra.run.smithi045.stdout:(8/24): python3-ceph-common-17.2.7-847.g45317f9 1.7 MB/s | 117 kB 00:00 2024-04-16T11:26:46.115 INFO:teuthology.orchestra.run.smithi045.stdout:(9/24): python3-cephfs-17.2.7-847.g45317f95.el9 1.4 MB/s | 156 kB 00:00 2024-04-16T11:26:46.149 INFO:teuthology.orchestra.run.smithi045.stdout:(10/24): python3-rados-17.2.7-847.g45317f95.el9 3.1 MB/s | 320 kB 00:00 2024-04-16T11:26:46.216 INFO:teuthology.orchestra.run.smithi045.stdout:(11/24): python3-rbd-17.2.7-847.g45317f95.el9.x 2.9 MB/s | 293 kB 00:00 2024-04-16T11:26:46.241 INFO:teuthology.orchestra.run.smithi045.stdout:(12/24): python3-rgw-17.2.7-847.g45317f95.el9.x 1.1 MB/s | 100 kB 00:00 2024-04-16T11:26:46.433 INFO:teuthology.orchestra.run.smithi045.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 302 kB/s | 44 kB 00:00 2024-04-16T11:26:46.458 INFO:teuthology.orchestra.run.smithi045.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 159 kB 00:00 2024-04-16T11:26:46.517 INFO:teuthology.orchestra.run.smithi044.stdout:(1/24): ceph-common-17.2.7-847.g45317f95.el9.x8 11 MB/s | 22 MB 00:02 2024-04-16T11:26:46.550 INFO:teuthology.orchestra.run.smithi045.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 661 kB 00:00 2024-04-16T11:26:46.559 INFO:teuthology.orchestra.run.smithi044.stdout:(2/24): ceph-selinux-17.2.7-847.g45317f95.el9.x 624 kB/s | 25 kB 00:00 2024-04-16T11:26:46.584 INFO:teuthology.orchestra.run.smithi045.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-04-16T11:26:46.609 INFO:teuthology.orchestra.run.smithi045.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 745 kB/s | 41 kB 00:00 2024-04-16T11:26:46.634 INFO:teuthology.orchestra.run.smithi044.stdout:(3/24): libcephfs2-17.2.7-847.g45317f95.el9.x86 8.4 MB/s | 639 kB 00:00 2024-04-16T11:26:46.701 INFO:teuthology.orchestra.run.smithi044.stdout:(4/24): libradosstriper1-17.2.7-847.g45317f95.e 6.8 MB/s | 457 kB 00:00 2024-04-16T11:26:46.818 INFO:teuthology.orchestra.run.smithi045.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 234 kB/s | 49 kB 00:00 2024-04-16T11:26:46.958 INFO:teuthology.orchestra.run.smithi045.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 831 kB/s | 309 kB 00:00 2024-04-16T11:26:47.043 INFO:teuthology.orchestra.run.smithi044.stdout:(5/24): librgw2-17.2.7-847.g45317f95.el9.x86_64 14 MB/s | 4.9 MB 00:00 2024-04-16T11:26:47.077 INFO:teuthology.orchestra.run.smithi045.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 262 kB/s | 67 kB 00:00 2024-04-16T11:26:47.093 INFO:teuthology.orchestra.run.smithi044.stdout:(6/24): python3-ceph-argparse-17.2.7-847.g45317 925 kB/s | 45 kB 00:00 2024-04-16T11:26:47.144 INFO:teuthology.orchestra.run.smithi044.stdout:(7/24): python3-ceph-common-17.2.7-847.g45317f9 2.3 MB/s | 117 kB 00:00 2024-04-16T11:26:47.194 INFO:teuthology.orchestra.run.smithi044.stdout:(8/24): python3-cephfs-17.2.7-847.g45317f95.el9 3.1 MB/s | 156 kB 00:00 2024-04-16T11:26:47.294 INFO:teuthology.orchestra.run.smithi044.stdout:(9/24): ceph-base-17.2.7-847.g45317f95.el9.x86_ 1.8 MB/s | 5.0 MB 00:02 2024-04-16T11:26:47.328 INFO:teuthology.orchestra.run.smithi044.stdout:(10/24): python3-rados-17.2.7-847.g45317f95.el9 2.4 MB/s | 320 kB 00:00 2024-04-16T11:26:47.378 INFO:teuthology.orchestra.run.smithi044.stdout:(11/24): python3-rgw-17.2.7-847.g45317f95.el9.x 1.9 MB/s | 100 kB 00:00 2024-04-16T11:26:47.428 INFO:teuthology.orchestra.run.smithi044.stdout:(12/24): python3-rbd-17.2.7-847.g45317f95.el9.x 2.2 MB/s | 293 kB 00:00 2024-04-16T11:26:47.594 INFO:teuthology.orchestra.run.smithi045.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-04-16T11:26:47.620 INFO:teuthology.orchestra.run.smithi044.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 287 kB/s | 44 kB 00:00 2024-04-16T11:26:47.645 INFO:teuthology.orchestra.run.smithi044.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 598 kB/s | 159 kB 00:00 2024-04-16T11:26:47.754 INFO:teuthology.orchestra.run.smithi044.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.9 MB/s | 661 kB 00:00 2024-04-16T11:26:47.786 INFO:teuthology.orchestra.run.smithi045.stdout:(22/24): librados2-17.2.7-847.g45317f95.el9.x86 4.3 MB/s | 3.1 MB 00:00 2024-04-16T11:26:47.788 INFO:teuthology.orchestra.run.smithi044.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 282 kB 00:00 2024-04-16T11:26:47.813 INFO:teuthology.orchestra.run.smithi044.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 744 kB/s | 41 kB 00:00 2024-04-16T11:26:47.928 INFO:teuthology.orchestra.run.smithi045.stdout:(23/24): librbd1-17.2.7-847.g45317f95.el9.x86_6 8.9 MB/s | 2.9 MB 00:00 2024-04-16T11:26:48.178 INFO:teuthology.orchestra.run.smithi045.stdout:(24/24): ceph-radosgw-17.2.7-847.g45317f95.el9. 3.0 MB/s | 14 MB 00:04 2024-04-16T11:26:48.187 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:26:48.187 INFO:teuthology.orchestra.run.smithi045.stdout:Total 12 MB/s | 58 MB 00:04 2024-04-16T11:26:48.189 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:26:48.240 INFO:teuthology.orchestra.run.smithi044.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 114 kB/s | 49 kB 00:00 2024-04-16T11:26:48.271 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:26:48.271 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:26:48.314 INFO:teuthology.orchestra.run.smithi044.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 588 kB/s | 309 kB 00:00 2024-04-16T11:26:48.356 INFO:teuthology.orchestra.run.smithi044.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 586 kB/s | 67 kB 00:00 2024-04-16T11:26:48.573 INFO:teuthology.orchestra.run.smithi044.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-04-16T11:26:48.816 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:26:48.816 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:26:48.848 INFO:teuthology.orchestra.run.smithi044.stdout:(22/24): librbd1-17.2.7-847.g45317f95.el9.x86_6 11 MB/s | 2.9 MB 00:00 2024-04-16T11:26:49.549 INFO:teuthology.orchestra.run.smithi044.stdout:(23/24): librados2-17.2.7-847.g45317f95.el9.x86 2.6 MB/s | 3.1 MB 00:01 2024-04-16T11:26:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:26:50.586 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-16T11:26:50.655 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T11:26:50.789 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T11:26:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T11:26:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T11:26:51.068 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T11:26:51.133 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-16T11:26:51.172 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-16T11:26:51.290 INFO:teuthology.orchestra.run.smithi044.stdout:(24/24): ceph-radosgw-17.2.7-847.g45317f95.el9. 2.1 MB/s | 14 MB 00:06 2024-04-16T11:26:51.304 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:26:51.304 INFO:teuthology.orchestra.run.smithi044.stdout:Total 8.2 MB/s | 58 MB 00:07 2024-04-16T11:26:51.305 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:26:51.360 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:26:51.360 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:26:51.725 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-16T11:26:51.751 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T11:26:51.901 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T11:26:51.906 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:26:51.906 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:26:51.945 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-16T11:26:51.999 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 10/26 2024-04-16T11:26:52.056 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T11:26:52.169 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T11:26:52.204 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T11:26:52.290 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T11:26:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-16T11:26:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-16T11:26:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-16T11:26:52.952 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-16T11:26:52.983 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T11:26:53.105 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T11:26:53.204 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 19/26 2024-04-16T11:26:53.243 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 20/26 2024-04-16T11:26:53.254 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:26:53.666 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-16T11:26:53.726 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T11:26:53.861 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T11:26:53.967 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T11:26:53.994 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T11:26:54.153 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T11:26:54.227 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-16T11:26:54.260 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-16T11:26:54.826 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-16T11:26:54.855 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T11:26:54.930 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T11:26:54.969 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-16T11:26:55.040 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 10/26 2024-04-16T11:26:55.094 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T11:26:55.208 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T11:26:55.242 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T11:26:55.371 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T11:26:55.447 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-16T11:26:55.498 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-16T11:26:55.545 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-16T11:26:55.946 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-16T11:26:55.979 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T11:26:56.110 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T11:26:56.149 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T11:26:56.190 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T11:26:56.209 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 19/26 2024-04-16T11:26:56.248 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 20/26 2024-04-16T11:26:56.832 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T11:26:56.877 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T11:26:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T11:26:57.022 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T11:26:57.022 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T11:26:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:26:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T11:26:59.161 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T11:26:59.424 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T11:27:00.073 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T11:27:00.216 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T11:27:00.406 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T11:27:00.406 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T11:27:00.406 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T11:27:00.406 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:00.467 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T11:27:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T11:27:08.510 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-04-16T11:27:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:09.973 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T11:27:10.158 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T11:27:10.158 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:10.158 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T11:27:10.158 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T11:27:10.159 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T11:27:10.159 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:10.163 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T11:27:10.286 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T11:27:10.287 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T11:27:12.017 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T11:27:12.017 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-04-16T11:27:12.018 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:12.023 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T11:27:12.023 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 1/26 2024-04-16T11:27:12.023 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 5/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 6/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 7/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 8/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 9/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 10/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-16T11:27:12.024 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-16T11:27:12.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 25/26 2024-04-16T11:27:12.730 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T11:27:12.730 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:12.730 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-04-16T11:27:12.730 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.730 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.730 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T11:27:12.731 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:12.732 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:13.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-04-16T11:27:13.466 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T11:27:13.614 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T11:27:13.614 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:13.614 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T11:27:13.614 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T11:27:13.614 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T11:27:13.614 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:13.619 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T11:27:13.738 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T11:27:13.738 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T11:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:34 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:14.284 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:14.285 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:14.285 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:27:14.285 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:14.285 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:17.2.7-847.g45317f95.el9 ceph 48 M 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:17.2.7-847.g45317f95.el9 ceph 163 k 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-04-16T11:27:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:14.287 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 49 M 2024-04-16T11:27:14.287 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 193 M 2024-04-16T11:27:14.287 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:27:14.630 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 984 kB/s | 217 kB 00:00 2024-04-16T11:27:14.689 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): libcephsqlite-17.2.7-847.g45317f95.el9.x 583 kB/s | 163 kB 00:00 2024-04-16T11:27:14.714 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 300 kB 00:00 2024-04-16T11:27:14.739 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-04-16T11:27:14.865 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-16T11:27:15.478 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T11:27:15.478 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 1/26 2024-04-16T11:27:15.478 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T11:27:15.478 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T11:27:15.478 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 5/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 6/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 7/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 8/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 9/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 10/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-16T11:27:15.479 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-16T11:27:15.480 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 25/26 2024-04-16T11:27:16.337 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T11:27:16.337 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T11:27:16.338 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:16.339 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:16.831 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-test 2024-04-16T11:27:17.722 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:00:36 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:17.893 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:17.2.7-847.g45317f95.el9 ceph 48 M 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T11:27:17.894 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:17.2.7-847.g45317f95.el9 ceph 163 k 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout:Install 6 Packages 2024-04-16T11:27:17.895 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:17.896 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 49 M 2024-04-16T11:27:17.896 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 193 M 2024-04-16T11:27:17.896 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:27:18.250 INFO:teuthology.orchestra.run.smithi044.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 784 kB/s | 217 kB 00:00 2024-04-16T11:27:18.333 INFO:teuthology.orchestra.run.smithi044.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 300 kB 00:00 2024-04-16T11:27:18.358 INFO:teuthology.orchestra.run.smithi044.stdout:(3/6): libcephsqlite-17.2.7-847.g45317f95.el9.x 422 kB/s | 163 kB 00:00 2024-04-16T11:27:18.383 INFO:teuthology.orchestra.run.smithi044.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-16T11:27:18.542 INFO:teuthology.orchestra.run.smithi044.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-04-16T11:27:18.781 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-17.2.7-847.g45317f95.el9.x86_6 11 MB/s | 48 MB 00:04 2024-04-16T11:27:18.788 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:18.788 INFO:teuthology.orchestra.run.smithi045.stdout:Total 11 MB/s | 49 MB 00:04 2024-04-16T11:27:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:27:18.888 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:27:18.888 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:27:19.344 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:27:19.345 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:27:20.052 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:27:20.135 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T11:27:20.178 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T11:27:20.227 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T11:27:20.292 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T11:27:20.318 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T11:27:20.876 INFO:teuthology.orchestra.run.smithi044.stdout:(6/6): ceph-test-17.2.7-847.g45317f95.el9.x86_6 16 MB/s | 48 MB 00:02 2024-04-16T11:27:20.882 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:20.883 INFO:teuthology.orchestra.run.smithi044.stdout:Total 16 MB/s | 49 MB 00:02 2024-04-16T11:27:20.927 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:27:20.985 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:27:20.985 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:27:21.456 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:27:21.456 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:27:22.157 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:27:22.264 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T11:27:22.305 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T11:27:22.356 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T11:27:22.428 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T11:27:22.479 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T11:27:25.442 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T11:27:25.545 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T11:27:27.558 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T11:27:27.708 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T11:27:27.715 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T11:27:27.716 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 1/6 2024-04-16T11:27:27.716 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2/6 2024-04-16T11:27:27.716 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T11:27:27.716 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T11:27:27.716 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T11:27:28.047 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T11:27:28.048 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:28.048 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:28.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-04-16T11:27:29.244 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:49 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:29.409 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:29.412 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-04-16T11:27:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:27:29.412 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:17.2.7-847.g45317f95.el9 ceph 6.5 k 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.1 M 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:17.2.7-847.g45317f95.el9 ceph 1.4 M 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 240 k 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.9 M 2024-04-16T11:27:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:17.2.7-847.g45317f95.el9 ceph 15 M 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T11:27:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T11:27:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T11:27:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T11:27:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:29.417 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:27:29.417 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-04-16T11:27:29.417 INFO:teuthology.orchestra.run.smithi045.stdout:Install 33 Packages 2024-04-16T11:27:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:29.418 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 27 M 2024-04-16T11:27:29.418 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 96 M 2024-04-16T11:27:29.418 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:27:30.122 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T11:27:30.122 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 1/6 2024-04-16T11:27:30.123 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2/6 2024-04-16T11:27:30.123 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T11:27:30.123 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T11:27:30.123 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T11:27:30.485 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T11:27:30.485 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: jq-1.6-16.el9.x86_64 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:30.486 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:30.560 INFO:teuthology.orchestra.run.smithi045.stdout:(1/33): ceph-17.2.7-847.g45317f95.el9.x86_64.rp 13 kB/s | 6.5 kB 00:00 2024-04-16T11:27:30.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph 2024-04-16T11:27:31.685 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:00:50 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:31.849 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:17.2.7-847.g45317f95.el9 ceph 6.5 k 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.1 M 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:17.2.7-847.g45317f95.el9 ceph 1.4 M 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 240 k 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.9 M 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:17.2.7-847.g45317f95.el9 ceph 15 M 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T11:27:31.852 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T11:27:31.853 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:27:31.854 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-04-16T11:27:31.855 INFO:teuthology.orchestra.run.smithi044.stdout:Install 33 Packages 2024-04-16T11:27:31.855 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:31.855 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 27 M 2024-04-16T11:27:31.855 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 96 M 2024-04-16T11:27:31.855 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:27:32.795 INFO:teuthology.orchestra.run.smithi044.stdout:(1/33): ceph-17.2.7-847.g45317f95.el9.x86_64.rp 12 kB/s | 6.5 kB 00:00 2024-04-16T11:27:33.152 INFO:teuthology.orchestra.run.smithi045.stdout:(2/33): ceph-mgr-17.2.7-847.g45317f95.el9.x86_6 468 kB/s | 1.4 MB 00:03 2024-04-16T11:27:34.728 INFO:teuthology.orchestra.run.smithi045.stdout:(3/33): ceph-mds-17.2.7-847.g45317f95.el9.x86_6 455 kB/s | 2.1 MB 00:04 2024-04-16T11:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout:(4/33): ceph-mgr-modules-core-17.2.7-847.g45317 1.0 MB/s | 240 kB 00:00 2024-04-16T11:27:35.045 INFO:teuthology.orchestra.run.smithi044.stdout:(2/33): ceph-mgr-17.2.7-847.g45317f95.el9.x86_6 518 kB/s | 1.4 MB 00:02 2024-04-16T11:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 919 kB/s | 144 kB 00:00 2024-04-16T11:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 774 kB/s | 32 kB 00:00 2024-04-16T11:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-16T11:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 588 kB/s | 39 kB 00:00 2024-04-16T11:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-04-16T11:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-04-16T11:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 103 kB 00:00 2024-04-16T11:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.0 MB/s | 124 kB 00:00 2024-04-16T11:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.5 MB/s | 187 kB 00:00 2024-04-16T11:27:35.779 INFO:teuthology.orchestra.run.smithi044.stdout:(3/33): ceph-mds-17.2.7-847.g45317f95.el9.x86_6 603 kB/s | 2.1 MB 00:03 2024-04-16T11:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 485 kB/s | 125 kB 00:00 2024-04-16T11:27:35.922 INFO:teuthology.orchestra.run.smithi044.stdout:(4/33): ceph-mgr-modules-core-17.2.7-847.g45317 1.7 MB/s | 240 kB 00:00 2024-04-16T11:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout:(15/33): ceph-mon-17.2.7-847.g45317f95.el9.x86_ 746 kB/s | 3.9 MB 00:05 2024-04-16T11:27:36.080 INFO:teuthology.orchestra.run.smithi044.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 924 kB/s | 144 kB 00:00 2024-04-16T11:27:36.113 INFO:teuthology.orchestra.run.smithi044.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 961 kB/s | 32 kB 00:00 2024-04-16T11:27:36.155 INFO:teuthology.orchestra.run.smithi044.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-16T11:27:36.197 INFO:teuthology.orchestra.run.smithi044.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 948 kB/s | 39 kB 00:00 2024-04-16T11:27:36.255 INFO:teuthology.orchestra.run.smithi044.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-04-16T11:27:36.331 INFO:teuthology.orchestra.run.smithi044.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-16T11:27:36.348 INFO:teuthology.orchestra.run.smithi045.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 78 kB/s | 43 kB 00:00 2024-04-16T11:27:36.381 INFO:teuthology.orchestra.run.smithi044.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-04-16T11:27:36.423 INFO:teuthology.orchestra.run.smithi044.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.0 MB/s | 124 kB 00:00 2024-04-16T11:27:36.465 INFO:teuthology.orchestra.run.smithi044.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.5 MB/s | 187 kB 00:00 2024-04-16T11:27:36.615 INFO:teuthology.orchestra.run.smithi044.stdout:(14/33): ceph-mon-17.2.7-847.g45317f95.el9.x86_ 1.0 MB/s | 3.9 MB 00:03 2024-04-16T11:27:36.657 INFO:teuthology.orchestra.run.smithi044.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 654 kB/s | 125 kB 00:00 2024-04-16T11:27:36.698 INFO:teuthology.orchestra.run.smithi045.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 252 kB/s | 172 kB 00:00 2024-04-16T11:27:36.732 INFO:teuthology.orchestra.run.smithi045.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 936 kB/s | 358 kB 00:00 2024-04-16T11:27:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 86 kB/s | 11 kB 00:00 2024-04-16T11:27:36.850 INFO:teuthology.orchestra.run.smithi045.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 152 kB/s | 18 kB 00:00 2024-04-16T11:27:36.941 INFO:teuthology.orchestra.run.smithi045.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 200 kB/s | 23 kB 00:00 2024-04-16T11:27:36.966 INFO:teuthology.orchestra.run.smithi045.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 167 kB/s | 19 kB 00:00 2024-04-16T11:27:37.016 INFO:teuthology.orchestra.run.smithi044.stdout:(16/33): ceph-osd-17.2.7-847.g45317f95.el9.x86_ 7.8 MB/s | 15 MB 00:01 2024-04-16T11:27:37.049 INFO:teuthology.orchestra.run.smithi044.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 100 kB/s | 43 kB 00:00 2024-04-16T11:27:37.058 INFO:teuthology.orchestra.run.smithi045.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 169 kB/s | 20 kB 00:00 2024-04-16T11:27:37.092 INFO:teuthology.orchestra.run.smithi045.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 372 kB/s | 46 kB 00:00 2024-04-16T11:27:37.150 INFO:teuthology.orchestra.run.smithi044.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-16T11:27:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 290 kB/s | 172 kB 00:00 2024-04-16T11:27:37.275 INFO:teuthology.orchestra.run.smithi044.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 142 kB/s | 18 kB 00:00 2024-04-16T11:27:37.301 INFO:teuthology.orchestra.run.smithi045.stdout:(25/33): ceph-osd-17.2.7-847.g45317f95.el9.x86_ 3.7 MB/s | 15 MB 00:04 2024-04-16T11:27:37.326 INFO:teuthology.orchestra.run.smithi045.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 296 kB/s | 79 kB 00:00 2024-04-16T11:27:37.359 INFO:teuthology.orchestra.run.smithi045.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-04-16T11:27:37.375 INFO:teuthology.orchestra.run.smithi044.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 234 kB/s | 23 kB 00:00 2024-04-16T11:27:37.400 INFO:teuthology.orchestra.run.smithi044.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 157 kB/s | 19 kB 00:00 2024-04-16T11:27:37.485 INFO:teuthology.orchestra.run.smithi045.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 570 kB/s | 90 kB 00:00 2024-04-16T11:27:37.492 INFO:teuthology.orchestra.run.smithi044.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 217 kB/s | 20 kB 00:00 2024-04-16T11:27:37.510 INFO:teuthology.orchestra.run.smithi045.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 238 kB/s | 36 kB 00:00 2024-04-16T11:27:37.518 INFO:teuthology.orchestra.run.smithi044.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 400 kB/s | 46 kB 00:00 2024-04-16T11:27:37.551 INFO:teuthology.orchestra.run.smithi044.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 671 kB/s | 358 kB 00:00 2024-04-16T11:27:37.660 INFO:teuthology.orchestra.run.smithi044.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-04-16T11:27:37.677 INFO:teuthology.orchestra.run.smithi045.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-16T11:27:37.685 INFO:teuthology.orchestra.run.smithi044.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 411 kB/s | 79 kB 00:00 2024-04-16T11:27:37.718 INFO:teuthology.orchestra.run.smithi044.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-16T11:27:37.719 INFO:teuthology.orchestra.run.smithi045.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.0 MB/s | 427 kB 00:00 2024-04-16T11:27:37.745 INFO:teuthology.orchestra.run.smithi045.stdout:(32/33): python3-portend-3.1.0-2.el9.noarch.rpm 37 kB/s | 16 kB 00:00 2024-04-16T11:27:37.761 INFO:teuthology.orchestra.run.smithi044.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 902 kB/s | 90 kB 00:00 2024-04-16T11:27:37.778 INFO:teuthology.orchestra.run.smithi044.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 391 kB/s | 36 kB 00:00 2024-04-16T11:27:37.794 INFO:teuthology.orchestra.run.smithi045.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 172 kB/s | 20 kB 00:00 2024-04-16T11:27:37.808 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:37.808 INFO:teuthology.orchestra.run.smithi045.stdout:Total 3.2 MB/s | 27 MB 00:08 2024-04-16T11:27:37.819 INFO:teuthology.orchestra.run.smithi044.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-16T11:27:37.852 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:27:37.870 INFO:teuthology.orchestra.run.smithi044.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 219 kB/s | 20 kB 00:00 2024-04-16T11:27:37.936 INFO:teuthology.orchestra.run.smithi044.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-04-16T11:27:37.943 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:27:37.943 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:27:37.944 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:37.945 INFO:teuthology.orchestra.run.smithi044.stdout:Total 4.4 MB/s | 27 MB 00:06 2024-04-16T11:27:37.987 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:27:38.076 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:27:38.076 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:27:38.370 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:27:38.370 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:27:38.493 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:27:38.494 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:27:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:27:39.815 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T11:27:39.864 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T11:27:39.903 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:27:39.916 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T11:27:39.966 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T11:27:40.001 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T11:27:40.047 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T11:27:40.066 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T11:27:40.165 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T11:27:40.192 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T11:27:40.224 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T11:27:40.332 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T11:27:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T11:27:40.400 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T11:27:40.417 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T11:27:40.469 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T11:27:40.499 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T11:27:40.538 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T11:27:40.560 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T11:27:40.602 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T11:27:40.624 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T11:27:40.669 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T11:27:40.751 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T11:27:40.883 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T11:27:40.948 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T11:27:41.000 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T11:27:41.045 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T11:27:41.049 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T11:27:41.086 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T11:27:41.114 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T11:27:41.152 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T11:27:41.198 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T11:27:41.213 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T11:27:41.234 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T11:27:41.252 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T11:27:41.286 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T11:27:41.336 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T11:27:41.357 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T11:27:41.394 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T11:27:41.401 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T11:27:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T11:27:41.436 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T11:27:41.477 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T11:27:41.522 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T11:27:41.569 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T11:27:41.571 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T11:27:41.619 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T11:27:41.704 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T11:27:41.751 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T11:27:41.783 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T11:27:41.916 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T11:27:41.975 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T11:27:42.120 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T11:27:42.141 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T11:27:42.280 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T11:27:42.326 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 28/33 2024-04-16T11:27:42.362 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T11:27:42.459 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 28/33 2024-04-16T11:27:42.465 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T11:27:42.465 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:42.465 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T11:27:42.465 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T11:27:42.465 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T11:27:42.466 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:42.493 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T11:27:42.569 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T11:27:42.569 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:42.569 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T11:27:42.569 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T11:27:42.569 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T11:27:42.569 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T11:27:44.060 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T11:27:44.061 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:44.061 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T11:27:44.061 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T11:27:44.061 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T11:27:44.061 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:44.128 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T11:27:44.211 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T11:27:44.212 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:44.212 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T11:27:44.212 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T11:27:44.212 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T11:27:44.212 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:44.488 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T11:27:44.566 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T11:27:44.566 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:44.566 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T11:27:44.566 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T11:27:44.567 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T11:27:44.567 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:44.653 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T11:27:44.726 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T11:27:44.726 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:44.726 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T11:27:44.727 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T11:27:44.727 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T11:27:44.727 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:44.797 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T11:27:44.890 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T11:27:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T11:27:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T11:27:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T11:27:44.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:44.958 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T11:27:44.962 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T11:27:45.032 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T11:27:45.032 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:45.032 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T11:27:45.032 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T11:27:45.032 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T11:27:45.032 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:45.066 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T11:27:45.066 INFO:teuthology.orchestra.run.smithi045.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T11:27:45.067 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:45.096 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T11:27:45.207 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T11:27:45.208 INFO:teuthology.orchestra.run.smithi044.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T11:27:45.208 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:46.825 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:17.2.7-847.g45317f95.el9.x86_64 1/33 2024-04-16T11:27:46.825 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2/33 2024-04-16T11:27:46.825 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 3/33 2024-04-16T11:27:46.825 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 4/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 5/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 6/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T11:27:46.826 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T11:27:46.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T11:27:46.828 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T11:27:46.828 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T11:27:46.829 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T11:27:46.829 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T11:27:46.829 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T11:27:46.829 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T11:27:46.829 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:17.2.7-847.g45317f95.el9.x86_64 1/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 3/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 4/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 5/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 6/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T11:27:46.932 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T11:27:46.933 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T11:27:46.934 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T11:27:46.935 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T11:27:48.041 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T11:27:48.042 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T11:27:48.043 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:48.044 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:48.070 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T11:27:48.070 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:48.070 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:27:48.070 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T11:27:48.071 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T11:27:48.072 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T11:27:48.073 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T11:27:48.073 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:48.073 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:48.422 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base 2024-04-16T11:27:48.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-04-16T11:27:49.306 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:08 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:49.368 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:09 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:49.415 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:27:49.471 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:49.472 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:27:49.472 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:49.477 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:27:49.533 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:49.535 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:27:49.535 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:49.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install cephadm 2024-04-16T11:27:49.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-04-16T11:27:50.415 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:09 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:50.483 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:10 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:50.580 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:50.580 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 82 k 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 82 k 2024-04-16T11:27:50.581 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 372 k 2024-04-16T11:27:50.582 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:27:50.648 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:50.648 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 82 k 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:27:50.649 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:50.650 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 82 k 2024-04-16T11:27:50.650 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 372 k 2024-04-16T11:27:50.650 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:27:51.246 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-17.2.7-847.g45317f95.el9.noarch.rpm 138 kB/s | 82 kB 00:00 2024-04-16T11:27:51.248 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:51.249 INFO:teuthology.orchestra.run.smithi045.stdout:Total 137 kB/s | 82 kB 00:00 2024-04-16T11:27:51.249 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:27:51.268 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:27:51.268 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:27:51.275 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:27:51.276 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:27:51.311 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm-17.2.7-847.g45317f95.el9.noarch.rpm 112 kB/s | 82 kB 00:00 2024-04-16T11:27:51.313 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:51.314 INFO:teuthology.orchestra.run.smithi044.stdout:Total 112 kB/s | 82 kB 00:00 2024-04-16T11:27:51.315 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:27:51.333 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:27:51.333 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:27:51.341 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:27:51.342 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:27:51.406 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:27:51.478 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:27:51.997 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.091 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.139 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.244 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.629 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.734 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:53.069 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T11:27:53.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:53.069 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:27:53.069 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:27:53.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:53.069 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:53.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T11:27:53.360 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T11:27:54.182 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:14 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:54.265 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:13 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:54.348 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-847.g45317f95.el9 ceph 137 k 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:27:54.349 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:54.350 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 137 k 2024-04-16T11:27:54.350 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 379 k 2024-04-16T11:27:54.350 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:27:54.434 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-847.g45317f95.el9 ceph 137 k 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:27:54.435 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:54.436 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 137 k 2024-04-16T11:27:54.436 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 379 k 2024-04-16T11:27:54.436 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:27:55.034 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-immutable-object-cache-17.2.7-847.g45317f9 230 kB/s | 137 kB 00:00 2024-04-16T11:27:55.036 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:55.037 INFO:teuthology.orchestra.run.smithi044.stdout:Total 229 kB/s | 137 kB 00:00 2024-04-16T11:27:55.037 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:27:55.064 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:27:55.064 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:27:55.136 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-17.2.7-847.g45317f9 175 kB/s | 137 kB 00:00 2024-04-16T11:27:55.138 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:27:55.139 INFO:teuthology.orchestra.run.smithi045.stdout:Total 174 kB/s | 137 kB 00:00 2024-04-16T11:27:55.139 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:27:55.167 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:27:55.167 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:27:55.167 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:27:55.167 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:27:55.268 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:27:55.269 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:27:55.395 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:27:55.473 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:27:55.477 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T11:27:55.537 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T11:27:55.564 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T11:27:55.564 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:55.564 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T11:27:55.564 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T11:27:55.618 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:27:55.618 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T11:27:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:56.825 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T11:27:56.825 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:56.825 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:27:56.825 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:56.825 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:56.825 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:56.919 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T11:27:56.919 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:56.919 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:27:56.920 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:27:56.920 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:56.920 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:57.132 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr 2024-04-16T11:27:57.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-04-16T11:27:58.031 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:17 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:58.142 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:27:58.198 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:58.200 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:27:58.200 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:27:58.212 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:18 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:58.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T11:27:58.321 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:27:58.377 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:58.378 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:27:58.378 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:27:58.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T11:27:59.171 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:18 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:27:59.331 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:19 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:27:59.341 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 3.4 M 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 24 k 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 15 k 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T11:27:59.342 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout:Install 7 Packages 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 3.7 M 2024-04-16T11:27:59.343 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 80 M 2024-04-16T11:27:59.344 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:27:59.501 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 3.4 M 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 24 k 2024-04-16T11:27:59.502 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 15 k 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout:Install 7 Packages 2024-04-16T11:27:59.503 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:27:59.504 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.7 M 2024-04-16T11:27:59.504 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 80 M 2024-04-16T11:27:59.504 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:00.183 INFO:teuthology.orchestra.run.smithi045.stdout:(1/7): ceph-prometheus-alerts-17.2.7-847.g45317 29 kB/s | 15 kB 00:00 2024-04-16T11:28:00.297 INFO:teuthology.orchestra.run.smithi044.stdout:(1/7): ceph-grafana-dashboards-17.2.7-847.g4531 49 kB/s | 24 kB 00:00 2024-04-16T11:28:00.324 INFO:teuthology.orchestra.run.smithi045.stdout:(2/7): ceph-grafana-dashboards-17.2.7-847.g4531 38 kB/s | 24 kB 00:00 2024-04-16T11:28:00.375 INFO:teuthology.orchestra.run.smithi045.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2024-04-16T11:28:00.514 INFO:teuthology.orchestra.run.smithi044.stdout:(2/7): ceph-prometheus-alerts-17.2.7-847.g45317 21 kB/s | 15 kB 00:00 2024-04-16T11:28:00.592 INFO:teuthology.orchestra.run.smithi045.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 153 kB/s | 41 kB 00:00 2024-04-16T11:28:00.642 INFO:teuthology.orchestra.run.smithi045.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 115 kB/s | 31 kB 00:00 2024-04-16T11:28:00.681 INFO:teuthology.orchestra.run.smithi044.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 24 kB/s | 9.0 kB 00:00 2024-04-16T11:28:00.848 INFO:teuthology.orchestra.run.smithi044.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 185 kB/s | 31 kB 00:00 2024-04-16T11:28:00.915 INFO:teuthology.orchestra.run.smithi044.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 102 kB/s | 41 kB 00:00 2024-04-16T11:28:01.042 INFO:teuthology.orchestra.run.smithi045.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 419 kB/s | 188 kB 00:00 2024-04-16T11:28:01.090 INFO:teuthology.orchestra.run.smithi044.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 784 kB/s | 188 kB 00:00 2024-04-16T11:28:01.565 INFO:teuthology.orchestra.run.smithi044.stdout:(7/7): ceph-mgr-dashboard-17.2.7-847.g45317f95. 1.9 MB/s | 3.4 MB 00:01 2024-04-16T11:28:01.574 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:01.575 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.6 MB/s | 3.7 MB 00:02 2024-04-16T11:28:01.575 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:01.605 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:01.605 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:01.739 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:01.740 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:02.148 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:02.226 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T11:28:02.282 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T11:28:02.383 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T11:28:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:(7/7): ceph-mgr-dashboard-17.2.7-847.g45317f95. 1.2 MB/s | 3.4 MB 00:02 2024-04-16T11:28:02.406 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:02.407 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.3 MB/s | 3.7 MB 00:02 2024-04-16T11:28:02.408 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:02.424 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T11:28:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:02.475 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 5/7 2024-04-16T11:28:02.577 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:02.577 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:03.003 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:03.077 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T11:28:03.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T11:28:03.234 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T11:28:03.268 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T11:28:03.319 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 5/7 2024-04-16T11:28:03.769 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 6/7 2024-04-16T11:28:03.820 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T11:28:04.613 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 6/7 2024-04-16T11:28:04.656 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T11:28:05.000 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T11:28:05.000 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 1/7 2024-04-16T11:28:05.001 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2/7 2024-04-16T11:28:05.001 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 3/7 2024-04-16T11:28:05.001 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T11:28:05.001 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T11:28:05.001 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T11:28:05.345 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T11:28:05.345 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:05.345 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:05.345 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:05.345 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:05.345 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:05.346 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T11:28:05.346 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T11:28:05.346 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T11:28:05.346 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T11:28:05.346 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:05.346 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:05.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 1/7 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2/7 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 3/7 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T11:28:05.859 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T11:28:06.251 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:06.252 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:06.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T11:28:06.576 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:06.747 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:06.748 INFO:teuthology.orchestra.run.smithi044.stdout:=========================================================================================== 2024-04-16T11:28:06.748 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:28:06.748 INFO:teuthology.orchestra.run.smithi044.stdout:=========================================================================================== 2024-04-16T11:28:06.748 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:06.748 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 7.4 M 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:06.749 INFO:teuthology.orchestra.run.smithi044.stdout:=========================================================================================== 2024-04-16T11:28:06.750 INFO:teuthology.orchestra.run.smithi044.stdout:Install 12 Packages 2024-04-16T11:28:06.750 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:06.750 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 37 M 2024-04-16T11:28:06.750 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 209 M 2024-04-16T11:28:06.750 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:07.349 INFO:teuthology.orchestra.run.smithi044.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 131 kB/s | 32 kB 00:00 2024-04-16T11:28:07.390 INFO:teuthology.orchestra.run.smithi044.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 465 kB/s | 17 kB 00:00 2024-04-16T11:28:07.481 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:27 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:07.507 INFO:teuthology.orchestra.run.smithi044.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.5 MB/s | 3.0 MB 00:00 2024-04-16T11:28:07.532 INFO:teuthology.orchestra.run.smithi044.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 272 kB/s | 34 kB 00:00 2024-04-16T11:28:07.653 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:07.654 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2024-04-16T11:28:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:28:07.654 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2024-04-16T11:28:07.654 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 7.4 M 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:07.655 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2024-04-16T11:28:07.656 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-04-16T11:28:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:07.656 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2024-04-16T11:28:07.656 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 209 M 2024-04-16T11:28:07.656 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:07.699 INFO:teuthology.orchestra.run.smithi044.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-04-16T11:28:07.808 INFO:teuthology.orchestra.run.smithi044.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-04-16T11:28:07.842 INFO:teuthology.orchestra.run.smithi044.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 217 kB 00:00 2024-04-16T11:28:08.068 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 123 kB/s | 32 kB 00:00 2024-04-16T11:28:08.110 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 477 kB/s | 17 kB 00:00 2024-04-16T11:28:08.217 INFO:teuthology.orchestra.run.smithi044.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-16T11:28:08.227 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-04-16T11:28:08.252 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 284 kB/s | 34 kB 00:00 2024-04-16T11:28:08.461 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-04-16T11:28:08.559 INFO:teuthology.orchestra.run.smithi044.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-16T11:28:08.561 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-04-16T11:28:08.592 INFO:teuthology.orchestra.run.smithi044.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 642 kB/s | 195 kB 00:00 2024-04-16T11:28:08.595 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-04-16T11:28:08.651 INFO:teuthology.orchestra.run.smithi044.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-04-16T11:28:08.970 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-16T11:28:09.137 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-16T11:28:09.179 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 941 kB/s | 195 kB 00:00 2024-04-16T11:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-04-16T11:28:09.576 INFO:teuthology.orchestra.run.smithi044.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-8 3.0 MB/s | 7.4 MB 00:02 2024-04-16T11:28:09.585 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:09.585 INFO:teuthology.orchestra.run.smithi044.stdout:Total 13 MB/s | 37 MB 00:02 2024-04-16T11:28:09.717 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:10.307 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:10.308 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T11:28:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T11:28:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T11:28:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T11:28:11.762 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T11:28:11.792 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T11:28:11.921 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T11:28:12.836 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T11:28:12.983 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T11:28:14.615 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T11:28:15.262 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-8 1.0 MB/s | 7.4 MB 00:07 2024-04-16T11:28:15.267 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:15.268 INFO:teuthology.orchestra.run.smithi045.stdout:Total 4.9 MB/s | 37 MB 00:07 2024-04-16T11:28:15.395 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:15.462 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:15.463 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:15.849 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T11:28:15.873 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T11:28:15.983 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:15.984 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:16.944 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:17.051 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T11:28:17.094 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T11:28:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T11:28:17.298 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T11:28:17.579 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T11:28:17.620 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T11:28:17.743 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 1/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T11:28:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T11:28:18.452 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T11:28:18.452 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T11:28:18.452 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T11:28:18.452 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T11:28:18.627 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T11:28:18.760 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T11:28:18.880 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T11:28:18.880 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:18.880 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:18.880 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:18.880 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:18.881 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:19.257 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-rook 2024-04-16T11:28:20.177 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:39 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:20.342 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:20.343 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-16T11:28:20.343 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:28:20.343 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-16T11:28:20.343 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 47 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T11:28:20.344 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout:Install 13 Packages 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:20.345 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 2.0 M 2024-04-16T11:28:20.346 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 26 M 2024-04-16T11:28:20.346 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:20.383 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T11:28:20.874 INFO:teuthology.orchestra.run.smithi044.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 92 kB/s | 24 kB 00:00 2024-04-16T11:28:20.899 INFO:teuthology.orchestra.run.smithi044.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 62 kB/s | 18 kB 00:00 2024-04-16T11:28:21.033 INFO:teuthology.orchestra.run.smithi044.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-04-16T11:28:21.058 INFO:teuthology.orchestra.run.smithi044.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-04-16T11:28:21.100 INFO:teuthology.orchestra.run.smithi044.stdout:(5/13): ceph-mgr-rook-17.2.7-847.g45317f95.el9. 97 kB/s | 47 kB 00:00 2024-04-16T11:28:21.134 INFO:teuthology.orchestra.run.smithi044.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 211 kB 00:00 2024-04-16T11:28:21.159 INFO:teuthology.orchestra.run.smithi044.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 428 kB/s | 43 kB 00:00 2024-04-16T11:28:21.434 INFO:teuthology.orchestra.run.smithi044.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 97 kB/s | 32 kB 00:00 2024-04-16T11:28:21.459 INFO:teuthology.orchestra.run.smithi044.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-04-16T11:28:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T11:28:21.646 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T11:28:21.685 INFO:teuthology.orchestra.run.smithi044.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 387 kB/s | 203 kB 00:00 2024-04-16T11:28:21.802 INFO:teuthology.orchestra.run.smithi044.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-16T11:28:22.127 INFO:teuthology.orchestra.run.smithi044.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-04-16T11:28:22.152 INFO:teuthology.orchestra.run.smithi044.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 192 kB/s | 90 kB 00:00 2024-04-16T11:28:22.158 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:22.159 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-04-16T11:28:22.160 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:22.205 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:22.205 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:22.371 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:22.373 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:23.053 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:23.256 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T11:28:23.319 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T11:28:23.394 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T11:28:23.432 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T11:28:23.487 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T11:28:23.627 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T11:28:23.772 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T11:28:23.849 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T11:28:24.018 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 1/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T11:28:24.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T11:28:24.436 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:24.437 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:24.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2024-04-16T11:28:24.855 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T11:28:24.905 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T11:28:24.939 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T11:28:24.992 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T11:28:25.015 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T11:28:25.723 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:25.888 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 47 k 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T11:28:25.889 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:25.890 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:25.891 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-16T11:28:25.891 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-04-16T11:28:25.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:25.891 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2024-04-16T11:28:25.891 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-04-16T11:28:25.891 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:26.031 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 1/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T11:28:26.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T11:28:26.408 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 126 kB/s | 24 kB 00:00 2024-04-16T11:28:26.434 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 83 kB/s | 18 kB 00:00 2024-04-16T11:28:26.483 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T11:28:26.500 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-04-16T11:28:26.505 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T11:28:26.505 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T11:28:26.506 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T11:28:26.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:26.507 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:26.534 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-04-16T11:28:26.551 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 899 kB/s | 43 kB 00:00 2024-04-16T11:28:26.701 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): ceph-mgr-rook-17.2.7-847.g45317f95.el9. 97 kB/s | 47 kB 00:00 2024-04-16T11:28:26.786 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T11:28:26.960 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2024-04-16T11:28:26.993 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-04-16T11:28:27.185 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 309 kB/s | 59 kB 00:00 2024-04-16T11:28:27.219 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 392 kB/s | 203 kB 00:00 2024-04-16T11:28:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 574 kB/s | 90 kB 00:00 2024-04-16T11:28:27.394 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-04-16T11:28:27.405 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:27.405 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-04-16T11:28:27.406 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:27.451 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:27.451 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:27.614 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:27.614 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:27.693 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:46 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:27.863 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:27.864 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-16T11:28:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:28:27.864 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-16T11:28:27.864 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:27.864 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 123 k 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout:Install 6 Packages 2024-04-16T11:28:27.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:27.866 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 6.8 M 2024-04-16T11:28:27.866 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 33 M 2024-04-16T11:28:27.866 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:28.295 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:28.505 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T11:28:28.573 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T11:28:28.588 INFO:teuthology.orchestra.run.smithi044.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-16T11:28:28.621 INFO:teuthology.orchestra.run.smithi044.stdout:(2/6): ceph-mgr-cephadm-17.2.7-847.g45317f95.el 279 kB/s | 123 kB 00:00 2024-04-16T11:28:28.648 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T11:28:28.684 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T11:28:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T11:28:28.862 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T11:28:28.922 INFO:teuthology.orchestra.run.smithi044.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 193 kB/s | 58 kB 00:00 2024-04-16T11:28:29.022 INFO:teuthology.orchestra.run.smithi044.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 514 kB/s | 51 kB 00:00 2024-04-16T11:28:29.032 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T11:28:29.100 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T11:28:29.155 INFO:teuthology.orchestra.run.smithi044.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 971 kB/s | 548 kB 00:00 2024-04-16T11:28:29.189 INFO:teuthology.orchestra.run.smithi044.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 246 kB/s | 248 kB 00:01 2024-04-16T11:28:29.193 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:29.194 INFO:teuthology.orchestra.run.smithi044.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-04-16T11:28:29.237 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:29.276 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:29.277 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:29.430 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:29.430 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:29.852 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:30.074 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T11:28:30.086 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T11:28:30.131 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T11:28:30.151 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T11:28:30.193 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T11:28:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T11:28:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T11:28:30.850 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T11:28:30.978 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T11:28:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T11:28:31.077 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 1/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T11:28:31.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T11:28:31.168 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T11:28:31.168 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T11:28:31.168 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T11:28:31.168 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T11:28:31.168 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T11:28:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:31.601 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:31.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T11:28:32.026 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T11:28:32.026 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/6 2024-04-16T11:28:32.026 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T11:28:32.026 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T11:28:32.026 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T11:28:32.026 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T11:28:32.372 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T11:28:32.373 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:32.373 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:32.681 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-fuse 2024-04-16T11:28:32.811 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:32.982 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 123 k 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T11:28:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-04-16T11:28:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:32.985 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.8 M 2024-04-16T11:28:32.985 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 33 M 2024-04-16T11:28:32.985 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:33.588 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:33.764 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 813 k 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:33.765 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:33.766 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:33.766 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-04-16T11:28:33.766 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:33.766 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 891 k 2024-04-16T11:28:33.766 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.6 M 2024-04-16T11:28:33.766 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:33.884 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): ceph-mgr-cephadm-17.2.7-847.g45317f95.el 307 kB/s | 123 kB 00:00 2024-04-16T11:28:34.075 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 256 kB/s | 79 kB 00:00 2024-04-16T11:28:34.158 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-fuse-17.2.7-847.g45317f95.el9.x86_6 2.0 MB/s | 813 kB 00:00 2024-04-16T11:28:34.162 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:34.163 INFO:teuthology.orchestra.run.smithi044.stdout:Total 2.2 MB/s | 891 kB 00:00 2024-04-16T11:28:34.163 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:34.218 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.9 MB/s | 5.8 MB 00:00 2024-04-16T11:28:34.237 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:34.237 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:34.251 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 322 kB/s | 248 kB 00:00 2024-04-16T11:28:34.338 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:34.338 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 248 kB/s | 58 kB 00:00 2024-04-16T11:28:34.476 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 227 kB/s | 51 kB 00:00 2024-04-16T11:28:34.510 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 882 kB/s | 548 kB 00:00 2024-04-16T11:28:34.516 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:34.517 INFO:teuthology.orchestra.run.smithi045.stdout:Total 4.4 MB/s | 6.8 MB 00:01 2024-04-16T11:28:34.561 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:34.599 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:34.599 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:34.635 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:34.748 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:34.749 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:34.813 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T11:28:34.906 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T11:28:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T11:28:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T11:28:35.818 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T11:28:35.819 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/2 2024-04-16T11:28:36.111 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T11:28:36.111 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:36.111 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:36.111 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T11:28:36.111 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:36.111 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T11:28:36.298 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T11:28:36.380 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T11:28:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T11:28:36.462 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados-devel 2024-04-16T11:28:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T11:28:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/6 2024-04-16T11:28:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T11:28:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T11:28:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T11:28:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T11:28:37.397 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:56 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:37.567 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:37.567 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:37.567 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-04-16T11:28:37.567 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:37.567 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 126 k 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:37.568 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 126 k 2024-04-16T11:28:37.569 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 448 k 2024-04-16T11:28:37.569 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:37.604 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:37.790 INFO:teuthology.orchestra.run.smithi044.stdout:librados-devel-17.2.7-847.g45317f95.el9.x86_64. 571 kB/s | 126 kB 00:00 2024-04-16T11:28:37.792 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:37.792 INFO:teuthology.orchestra.run.smithi044.stdout:Total 563 kB/s | 126 kB 00:00 2024-04-16T11:28:37.793 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:37.812 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:37.812 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:37.931 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:37.932 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:37.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-04-16T11:28:38.146 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:38.267 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:38.841 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:38.859 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:58 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:39.034 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 813 k 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:39.035 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:39.036 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:39.036 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-04-16T11:28:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:39.036 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 891 k 2024-04-16T11:28:39.036 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.6 M 2024-04-16T11:28:39.036 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:39.120 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:39.120 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:39.120 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:39.120 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:39.121 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:39.121 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:39.171 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 587 kB/s | 79 kB 00:00 2024-04-16T11:28:39.403 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs2 2024-04-16T11:28:39.971 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-17.2.7-847.g45317f95.el9.x86_6 870 kB/s | 813 kB 00:00 2024-04-16T11:28:39.975 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:39.976 INFO:teuthology.orchestra.run.smithi045.stdout:Total 948 kB/s | 891 kB 00:00 2024-04-16T11:28:39.977 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:40.052 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:40.052 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:40.153 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:40.153 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:40.316 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:59 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:40.428 INFO:teuthology.orchestra.run.smithi044.stdout:Package libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:40.428 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:40.484 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:40.486 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:40.486 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:40.554 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs-devel 2024-04-16T11:28:40.596 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T11:28:40.680 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T11:28:41.460 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:00 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:41.629 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:41.629 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:28:41.630 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:41.631 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 25 k 2024-04-16T11:28:41.631 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 110 k 2024-04-16T11:28:41.631 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:41.636 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T11:28:41.636 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/2 2024-04-16T11:28:41.987 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T11:28:41.988 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:41.988 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:41.988 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T11:28:41.988 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:41.988 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:42.144 INFO:teuthology.orchestra.run.smithi044.stdout:libcephfs-devel-17.2.7-847.g45317f95.el9.x86_64 49 kB/s | 25 kB 00:00 2024-04-16T11:28:42.146 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:42.146 INFO:teuthology.orchestra.run.smithi044.stdout:Total 48 kB/s | 25 kB 00:00 2024-04-16T11:28:42.147 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:42.159 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:42.159 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:42.166 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:42.167 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:42.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-04-16T11:28:42.315 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:42.478 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:42.955 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:43.221 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:03 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:43.229 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:43.230 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:43.230 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:43.230 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:43.230 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:43.230 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:43.390 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 126 k 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:43.391 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:28:43.392 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 126 k 2024-04-16T11:28:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 448 k 2024-04-16T11:28:43.392 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:43.422 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados2 2024-04-16T11:28:43.994 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-17.2.7-847.g45317f95.el9.x86_64. 210 kB/s | 126 kB 00:00 2024-04-16T11:28:43.996 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:43.996 INFO:teuthology.orchestra.run.smithi045.stdout:Total 209 kB/s | 126 kB 00:00 2024-04-16T11:28:43.997 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:44.017 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:44.017 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:44.139 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:44.140 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:44.330 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:03 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:44.449 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:44.463 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:44.506 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:44.507 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:44.507 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:44.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librbd1 2024-04-16T11:28:44.998 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:45.325 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:45.325 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:45.325 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:45.325 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:45.325 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:45.326 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:45.476 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:04 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:45.596 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:45.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-04-16T11:28:45.652 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:45.654 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:45.654 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:45.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rados 2024-04-16T11:28:46.536 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:06 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:46.599 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:05 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:46.648 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:46.704 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:46.706 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:46.706 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:46.712 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:46.773 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:46.774 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:46.774 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:46.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-04-16T11:28:46.845 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rgw 2024-04-16T11:28:47.672 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:07 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:47.739 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:06 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:47.841 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:47.841 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:47.841 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:28:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:47.843 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 25 k 2024-04-16T11:28:47.843 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 110 k 2024-04-16T11:28:47.843 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:47.853 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:47.913 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:47.915 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:47.915 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:47.988 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-cephfs 2024-04-16T11:28:48.340 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-17.2.7-847.g45317f95.el9.x86_64 50 kB/s | 25 kB 00:00 2024-04-16T11:28:48.341 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:48.342 INFO:teuthology.orchestra.run.smithi045.stdout:Total 50 kB/s | 25 kB 00:00 2024-04-16T11:28:48.342 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:48.355 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:48.355 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:48.361 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:48.363 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:48.503 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:48.649 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:48.878 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:07 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:48.989 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:49.047 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:49.048 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:49.049 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:49.088 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:49.222 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rbd 2024-04-16T11:28:49.372 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:49.372 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:49.372 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:49.372 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:49.372 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:49.373 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:49.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-04-16T11:28:50.099 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:09 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:50.213 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:50.273 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:50.275 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-16T11:28:50.275 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:50.353 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-fuse 2024-04-16T11:28:50.499 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:10 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:50.618 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:50.677 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:50.678 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:50.679 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:50.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-04-16T11:28:51.208 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:10 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:51.385 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:51.386 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:51.386 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-04-16T11:28:51.386 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 82 k 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:28:51.387 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:51.388 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 82 k 2024-04-16T11:28:51.388 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 222 k 2024-04-16T11:28:51.388 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:51.644 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:11 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:51.762 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:51.822 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:51.823 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:51.823 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:51.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-04-16T11:28:51.989 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-fuse-17.2.7-847.g45317f95.el9.x86_64.rpm 136 kB/s | 82 kB 00:00 2024-04-16T11:28:51.991 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:51.991 INFO:teuthology.orchestra.run.smithi044.stdout:Total 135 kB/s | 82 kB 00:00 2024-04-16T11:28:51.992 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:52.016 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:52.017 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:52.122 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:52.122 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:52.328 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:52.467 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:52.791 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:12 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:52.904 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:52.964 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:52.965 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:52.965 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:53.030 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:53.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-04-16T11:28:53.295 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:53.295 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:53.295 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:53.296 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:53.296 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:53.296 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:53.632 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-mirror 2024-04-16T11:28:53.924 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:13 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:54.036 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:54.096 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:54.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-04-16T11:28:54.544 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:13 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:54.717 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:28:54.718 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:54.719 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 2.9 M 2024-04-16T11:28:54.719 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 12 M 2024-04-16T11:28:54.719 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:55.078 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:15 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:55.247 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:55.248 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:55.248 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:55.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-04-16T11:28:55.909 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-mirror-17.2.7-847.g45317f95.el9.x86_64.rpm 2.5 MB/s | 2.9 MB 00:01 2024-04-16T11:28:55.911 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:55.912 INFO:teuthology.orchestra.run.smithi044.stdout:Total 2.5 MB/s | 2.9 MB 00:01 2024-04-16T11:28:55.912 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:28:55.944 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:28:55.944 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:28:56.072 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:28:56.073 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:28:56.199 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:16 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:56.313 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T11:28:56.373 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:56.374 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-04-16T11:28:56.375 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:56.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-04-16T11:28:56.722 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:28:56.784 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:56.866 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:56.866 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:28:56.866 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T11:28:56.866 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T11:28:56.866 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T11:28:56.866 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:57.341 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:17 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:28:57.517 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 82 k 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:57.518 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:28:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:28:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:28:57.519 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 82 k 2024-04-16T11:28:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 222 k 2024-04-16T11:28:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:28:57.734 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-17.2.7-847.g45317f95.el9.x86_64.rpm 379 kB/s | 82 kB 00:00 2024-04-16T11:28:57.736 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:57.737 INFO:teuthology.orchestra.run.smithi045.stdout:Total 374 kB/s | 82 kB 00:00 2024-04-16T11:28:57.737 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:28:57.761 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:28:57.761 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:28:57.866 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:28:57.866 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:28:58.070 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:28:58.184 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:58.310 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:58.310 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:58.310 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:28:58.310 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:58.310 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:58.310 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:28:58.636 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:58.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-nbd 2024-04-16T11:28:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:28:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:58.926 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:28:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:28:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:28:58.926 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:28:59.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-04-16T11:28:59.579 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:18 ago on Tue 16 Apr 2024 11:26:41 AM UTC. 2024-04-16T11:28:59.755 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 165 k 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-04-16T11:28:59.756 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:28:59.757 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 165 k 2024-04-16T11:28:59.757 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 477 k 2024-04-16T11:28:59.757 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-16T11:28:59.996 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-nbd-17.2.7-847.g45317f95.el9.x86_64.rpm 693 kB/s | 165 kB 00:00 2024-04-16T11:28:59.997 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:28:59.998 INFO:teuthology.orchestra.run.smithi044.stdout:Total 684 kB/s | 165 kB 00:00 2024-04-16T11:28:59.999 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-16T11:29:00.025 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-16T11:29:00.025 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-16T11:29:00.128 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-16T11:29:00.128 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-16T11:29:00.161 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:20 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:29:00.335 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:29:00.336 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:29:00.336 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.9 M 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 12 M 2024-04-16T11:29:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:29:00.343 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-16T11:29:00.462 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:01.057 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:01.402 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:01.403 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:29:01.403 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-16T11:29:01.403 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:29:01.403 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:29:01.403 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-16T11:29:01.748 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-17.2.7-847.g45317f95.el9.x86_64.rpm 2.1 MB/s | 2.9 MB 00:01 2024-04-16T11:29:01.750 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:29:01.750 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.1 MB/s | 2.9 MB 00:01 2024-04-16T11:29:01.751 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:29:01.769 DEBUG:teuthology.parallel:result is None 2024-04-16T11:29:01.782 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:29:01.783 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:29:01.905 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:29:01.905 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:29:02.439 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:29:02.506 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:02.587 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:02.587 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T11:29:02.587 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T11:29:02.587 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T11:29:02.587 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T11:29:02.587 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:04.068 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:04.068 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:04.068 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:29:04.069 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:29:04.069 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:04.069 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:29:04.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-04-16T11:29:05.291 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:25 ago on Tue 16 Apr 2024 11:26:40 AM UTC. 2024-04-16T11:29:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-16T11:29:05.468 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:29:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 165 k 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 165 k 2024-04-16T11:29:05.469 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 477 k 2024-04-16T11:29:05.470 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-16T11:29:05.710 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-17.2.7-847.g45317f95.el9.x86_64.rpm 685 kB/s | 165 kB 00:00 2024-04-16T11:29:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:29:05.712 INFO:teuthology.orchestra.run.smithi045.stdout:Total 675 kB/s | 165 kB 00:00 2024-04-16T11:29:05.713 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-16T11:29:05.740 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-16T11:29:05.740 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-16T11:29:05.842 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-16T11:29:05.843 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-16T11:29:06.044 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-16T11:29:06.189 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:06.746 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:07.039 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T11:29:07.039 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:07.039 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-16T11:29:07.039 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T11:29:07.039 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:29:07.039 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-16T11:29:07.333 DEBUG:teuthology.parallel:result is None 2024-04-16T11:29:07.333 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:29:07.333 INFO:teuthology.packaging:ref: None 2024-04-16T11:29:07.334 INFO:teuthology.packaging:tag: None 2024-04-16T11:29:07.334 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T11:29:07.334 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:29:07.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T11:29:07.479 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T11:29:07.562 INFO:teuthology.orchestra.run.smithi044.stdout:17.2.7-847.g45317f95.el9 2024-04-16T11:29:07.563 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-847.g45317f95.el9 2024-04-16T11:29:07.563 INFO:teuthology.task.install:The correct ceph version 17.2.7-847.g45317f95 is installed. 2024-04-16T11:29:07.565 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T11:29:07.565 INFO:teuthology.packaging:ref: None 2024-04-16T11:29:07.565 INFO:teuthology.packaging:tag: None 2024-04-16T11:29:07.565 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T11:29:07.565 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:29:07.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T11:29:07.778 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T11:29:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:17.2.7-847.g45317f95.el9 2024-04-16T11:29:07.852 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-847.g45317f95.el9 2024-04-16T11:29:07.853 INFO:teuthology.task.install:The correct ceph version 17.2.7-847.g45317f95 is installed. 2024-04-16T11:29:07.854 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T11:29:07.855 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:29:07.855 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T11:29:07.965 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:29:07.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T11:29:08.077 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T11:29:08.077 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:29:08.077 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T11:29:08.186 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T11:29:08.282 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:29:08.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T11:29:08.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T11:29:08.522 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T11:29:08.522 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:29:08.522 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T11:29:08.627 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T11:29:08.768 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:29:08.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T11:29:08.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T11:29:09.015 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T11:29:09.015 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:29:09.015 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T11:29:09.118 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T11:29:09.227 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:29:09.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T11:29:09.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T11:29:09.432 INFO:teuthology.run_tasks:Running task print... 2024-04-16T11:29:09.442 INFO:teuthology.task.print:**** done install task... 2024-04-16T11:29:09.442 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-16T11:29:09.840 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-16T11:29:09.840 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 2024-04-16T11:29:09.841 INFO:tasks.cephadm:Cluster fsid is 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:29:09.841 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-16T11:29:09.841 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-16T11:29:09.841 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi044': '172.21.15.44', 'mon.smithi045': '172.21.15.45'} 2024-04-16T11:29:09.841 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-16T11:29:09.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2024-04-16T11:29:09.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-04-16T11:29:10.055 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-16T11:29:10.056 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T11:29:10.438 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 16 11:29 /home/ubuntu/cephtest/cephadm 2024-04-16T11:29:10.438 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T11:29:10.685 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 16 11:29 /home/ubuntu/cephtest/cephadm 2024-04-16T11:29:10.685 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T11:29:10.750 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T11:29:10.819 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 on all hosts... 2024-04-16T11:29:10.819 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 pull 2024-04-16T11:29:10.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 pull 2024-04-16T11:29:12.022 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T11:29:12.201 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T11:30:04.592 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T11:30:04.592 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T11:30:04.592 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe", 2024-04-16T11:30:04.593 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-04-16T11:30:04.593 INFO:teuthology.orchestra.run.smithi045.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77" 2024-04-16T11:30:04.593 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-16T11:30:04.593 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe", 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77" 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-16T11:30:10.882 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:30:10.913 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2024-04-16T11:30:11.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-04-16T11:30:11.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2024-04-16T11:30:11.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-04-16T11:30:11.322 INFO:tasks.cephadm:Writing seed config... 2024-04-16T11:30:11.348 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-16T11:30:11.348 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-16T11:30:11.348 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-16T11:30:11.348 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-16T11:30:11.348 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-16T11:30:11.348 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-16T11:30:11.349 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-16T11:30:11.350 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-16T11:30:11.351 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-16T11:30:11.352 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-16T11:30:11.352 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-16T11:30:11.352 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:30:11.353 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-16T11:30:11.413 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8b0307a8-fbe4-11ee-bc8f-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-16T11:30:11.413 DEBUG:teuthology.orchestra.run.smithi044:mon.smithi044> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service 2024-04-16T11:30:11.460 INFO:tasks.cephadm:Bootstrapping... 2024-04-16T11:30:11.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 -v bootstrap --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.44 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-16T11:30:11.777 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-16T11:30:11.777 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3', '-v', 'bootstrap', '--fsid', '8b0307a8-fbe4-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.44', '--skip-admin-label'] 2024-04-16T11:30:11.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T11:30:11.838 INFO:teuthology.orchestra.run.smithi044.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-16T11:30:11.838 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying podman|docker is present... 2024-04-16T11:30:11.898 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T11:30:11.898 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying lvm2 is present... 2024-04-16T11:30:11.898 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying time synchronization is in place... 2024-04-16T11:30:11.915 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T11:30:11.915 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T11:30:11.934 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T11:30:11.934 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-16T11:30:11.956 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-04-16T11:30:11.976 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-04-16T11:30:11.976 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-04-16T11:30:11.977 INFO:teuthology.orchestra.run.smithi044.stdout:Repeating the final host check... 2024-04-16T11:30:12.032 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T11:30:12.032 INFO:teuthology.orchestra.run.smithi044.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-16T11:30:12.032 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl is present 2024-04-16T11:30:12.032 INFO:teuthology.orchestra.run.smithi044.stdout:lvcreate is present 2024-04-16T11:30:12.048 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T11:30:12.049 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T11:30:12.066 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T11:30:12.066 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-16T11:30:12.085 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-04-16T11:30:12.104 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-04-16T11:30:12.104 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-04-16T11:30:12.104 INFO:teuthology.orchestra.run.smithi044.stdout:Host looks OK 2024-04-16T11:30:12.104 INFO:teuthology.orchestra.run.smithi044.stdout:Cluster fsid: 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:12.104 INFO:teuthology.orchestra.run.smithi044.stdout:Acquiring lock 140573355344992 on /run/cephadm/8b0307a8-fbe4-11ee-bc8f-c7b262605968.lock 2024-04-16T11:30:12.105 INFO:teuthology.orchestra.run.smithi044.stdout:Lock 140573355344992 acquired on /run/cephadm/8b0307a8-fbe4-11ee-bc8f-c7b262605968.lock 2024-04-16T11:30:12.106 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 3300 ... 2024-04-16T11:30:12.106 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 6789 ... 2024-04-16T11:30:12.107 INFO:teuthology.orchestra.run.smithi044.stdout:Base mon IP(s) is [172.21.15.44:3300, 172.21.15.44:6789], mon addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-04-16T11:30:12.120 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.44 metric 100 2024-04-16T11:30:12.120 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2024-04-16T11:30:12.132 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-16T11:30:12.132 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-16T11:30:12.132 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-16T11:30:12.143 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-16T11:30:12.144 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-16T11:30:12.144 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T11:30:12.144 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-16T11:30:12.144 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-16T11:30:12.144 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T11:30:12.146 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-04-16T11:30:12.146 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-04-16T11:30:12.147 INFO:teuthology.orchestra.run.smithi044.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-16T11:30:12.149 INFO:teuthology.orchestra.run.smithi044.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-16T11:30:12.151 INFO:teuthology.orchestra.run.smithi044.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Getting image source signatures 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:0ca96691e690249b227cbeb34b4e7fdd96c9d73da309c97947ab03def7d1129d 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying config sha256:b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe 2024-04-16T11:30:12.838 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-16T11:30:14.038 INFO:teuthology.orchestra.run.smithi044.stdout:ceph: stdout ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable) 2024-04-16T11:30:14.038 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph version: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable) 2024-04-16T11:30:14.038 INFO:teuthology.orchestra.run.smithi044.stdout:Extracting ceph user uid/gid from container image... 2024-04-16T11:30:15.241 INFO:teuthology.orchestra.run.smithi044.stdout:stat: stdout 167 167 2024-04-16T11:30:15.241 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial keys... 2024-04-16T11:30:16.491 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQBHYR5mgVfZLhAAOTHBDX63p/HNFxVzxJ7sjw== 2024-04-16T11:30:17.654 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQBIYR5mppiEOxAAsZXAckxoIHMx7RJfxPn8LQ== 2024-04-16T11:30:18.920 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQBKYR5mbz7sCRAAaE+VPHHhDy/NNf+gfzXTog== 2024-04-16T11:30:18.922 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial monmap... 2024-04-16T11:30:20.017 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool for smithi044 [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = octopus 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: set fsid to 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:30:20.018 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mon... 2024-04-16T11:30:21.329 INFO:teuthology.orchestra.run.smithi044.stdout:create mon.smithi044 on 2024-04-16T11:30:21.697 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-16T11:30:21.942 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-16T11:30:22.213 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968.target. 2024-04-16T11:30:22.213 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968.target. 2024-04-16T11:30:22.512 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044 2024-04-16T11:30:22.512 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service: Unit ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service not loaded. 2024-04-16T11:30:22.821 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968.target.wants/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service → /etc/systemd/system/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@.service. 2024-04-16T11:30:23.536 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 podman[28341]: 2024-04-16 11:30:23.257914335 +0000 UTC m=+0.144327272 container create c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T11:30:23.536 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 podman[28341]: 2024-04-16 11:30:23.53679065 +0000 UTC m=+0.423203630 container init c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T11:30:23.667 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T11:30:23.668 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-16T11:30:23.686 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T11:30:23.687 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-16T11:30:23.687 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-16T11:30:23.687 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T11:30:23.687 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon to start... 2024-04-16T11:30:23.687 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon... 2024-04-16T11:30:23.836 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 podman[28341]: 2024-04-16 11:30:23.547942334 +0000 UTC m=+0.434355278 container start c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-16T11:30:23.836 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 bash[28341]: c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 2024-04-16T11:30:23.836 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 ceph-mon[28360]: mkfs 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:23.837 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 ceph-mon[28360]: mon.smithi044 is new leader, mons smithi044 in quorum (ranks 0) 2024-04-16T11:30:23.837 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:23 smithi044 systemd[1]: Started Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:30:24.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:24 smithi044 ceph-mon[28360]: mon.smithi044 is new leader, mons smithi044 in quorum (ranks 0) 2024-04-16T11:30:24.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:24 smithi044 ceph-mon[28360]: monmap e1: 1 mons at {smithi044=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T11:30:24.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:24 smithi044 ceph-mon[28360]: fsmap 2024-04-16T11:30:24.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:24 smithi044 ceph-mon[28360]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T11:30:24.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:24 smithi044 ceph-mon[28360]: mgrmap e1: no daemons active 2024-04-16T11:30:24.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:24 smithi044 ceph-mon[28360]: from='client.? 172.21.15.44:0/206511660' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-16T11:30:24.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout cluster: 2024-04-16T11:30:24.989 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout id: 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:24.989 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-16T11:30:24.989 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:24.989 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout services: 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi044 (age 0.740732s) 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout data: 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pgs: 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.270+0000 7fab41000700 1 Processor -- start 2024-04-16T11:30:24.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.270+0000 7fab41000700 1 -- start start 2024-04-16T11:30:24.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.271+0000 7fab41000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c077280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:24.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.271+0000 7fab41000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c079040 con 0x7fab3c078c30 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.272+0000 7fab39e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c077280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.272+0000 7fab39e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c077280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55504/0 (socket says 172.21.15.44:55504) 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.272+0000 7fab39e00700 1 -- 172.21.15.44:0/3534050305 learned_addr learned my addr 172.21.15.44:0/3534050305 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.272+0000 7fab39e00700 1 -- 172.21.15.44:0/3534050305 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c079180 con 0x7fab3c078c30 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.273+0000 7fab39e00700 1 --2- 172.21.15.44:0/3534050305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c077280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0098e0 tx=0x7fab2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141c38d4c13dc2da server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.273+0000 7fab38a00700 1 -- 172.21.15.44:0/3534050305 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c00b860 con 0x7fab3c078c30 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.273+0000 7fab38a00700 1 -- 172.21.15.44:0/3534050305 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c00b9c0 con 0x7fab3c078c30 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.273+0000 7fab38a00700 1 -- 172.21.15.44:0/3534050305 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c00bc90 con 0x7fab3c078c30 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 -- 172.21.15.44:0/3534050305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 msgr2=0x7fab3c077280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:24.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 --2- 172.21.15.44:0/3534050305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c077280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0098e0 tx=0x7fab2c009250 comp rx=0 tx=0).stop 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 -- 172.21.15.44:0/3534050305 shutdown_connections 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 --2- 172.21.15.44:0/3534050305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c077280 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 -- 172.21.15.44:0/3534050305 >> 172.21.15.44:0/3534050305 conn(0x7fab3c0fe120 msgr2=0x7fab3c100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 -- 172.21.15.44:0/3534050305 shutdown_connections 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.274+0000 7fab41000700 1 -- 172.21.15.44:0/3534050305 wait complete. 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.275+0000 7fab41000700 1 Processor -- start 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.275+0000 7fab41000700 1 -- start start 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.276+0000 7fab41000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.276+0000 7fab41000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c1134b0 con 0x7fab3c078c30 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.276+0000 7fab39e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.276+0000 7fab39e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55506/0 (socket says 172.21.15.44:55506) 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.276+0000 7fab39e00700 1 -- 172.21.15.44:0/206511660 learned_addr learned my addr 172.21.15.44:0/206511660 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:24.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.277+0000 7fab39e00700 1 -- 172.21.15.44:0/206511660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0094f0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.277+0000 7fab39e00700 1 --2- 172.21.15.44:0/206511660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c112f70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009290 tx=0x7fab2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.277+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c01c5c0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.277+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c01c740 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.277+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c1136b0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.277+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c01ca50 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.278+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c114220 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.278+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fab2c01cbb0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.278+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fab2c01adb0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.279+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c05f5f0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.282+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fab2c01a450 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.362+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fab3c114cf0 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.363+0000 7fab32000700 1 -- 172.21.15.44:0/206511660 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fab2c025440 con 0x7fab3c078c30 2024-04-16T11:30:24.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 msgr2=0x7fab3c112f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 --2- 172.21.15.44:0/206511660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c112f70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009290 tx=0x7fab2c016000 comp rx=0 tx=0).stop 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 shutdown_connections 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 --2- 172.21.15.44:0/206511660 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab3c078c30 0x7fab3c112f70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 >> 172.21.15.44:0/206511660 conn(0x7fab3c0fe120 msgr2=0x7fab3c109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 shutdown_connections 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:24.365+0000 7fab41000700 1 -- 172.21.15.44:0/206511660 wait complete. 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:mon is available 2024-04-16T11:30:25.000 INFO:teuthology.orchestra.run.smithi044.stdout:Assimilating anything we can from ceph.conf... 2024-04-16T11:30:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:25 smithi044 ceph-mon[28360]: from='client.? 172.21.15.44:0/2193695845' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T11:30:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:25 smithi044 ceph-mon[28360]: from='client.? 172.21.15.44:0/2193695845' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T11:30:26.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T11:30:26.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.574+0000 7f0a42c00700 1 Processor -- start 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.575+0000 7f0a42c00700 1 -- start start 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.575+0000 7f0a42c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c07ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.575+0000 7f0a42c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c07d420 con 0x7f0a3c07ca40 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.576+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c07ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.576+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c07ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55512/0 (socket says 172.21.15.44:55512) 2024-04-16T11:30:26.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.576+0000 7f0a3be00700 1 -- 172.21.15.44:0/1475471658 learned_addr learned my addr 172.21.15.44:0/1475471658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.576+0000 7f0a3be00700 1 -- 172.21.15.44:0/1475471658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c07d560 con 0x7f0a3c07ca40 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.577+0000 7f0a3be00700 1 --2- 172.21.15.44:0/1475471658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c07ce50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0098e0 tx=0x7f0a2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482035fd54f1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.577+0000 7f0a3aa00700 1 -- 172.21.15.44:0/1475471658 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a2c00b860 con 0x7f0a3c07ca40 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.577+0000 7f0a3aa00700 1 -- 172.21.15.44:0/1475471658 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c00b9c0 con 0x7f0a3c07ca40 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.577+0000 7f0a3aa00700 1 -- 172.21.15.44:0/1475471658 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a2c00bc90 con 0x7f0a3c07ca40 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.577+0000 7f0a42c00700 1 -- 172.21.15.44:0/1475471658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 msgr2=0x7f0a3c07ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.577+0000 7f0a42c00700 1 --2- 172.21.15.44:0/1475471658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c07ce50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0098e0 tx=0x7f0a2c009250 comp rx=0 tx=0).stop 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.578+0000 7f0a42c00700 1 -- 172.21.15.44:0/1475471658 shutdown_connections 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.578+0000 7f0a42c00700 1 --2- 172.21.15.44:0/1475471658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c07ce50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.578+0000 7f0a42c00700 1 -- 172.21.15.44:0/1475471658 >> 172.21.15.44:0/1475471658 conn(0x7f0a3c078350 msgr2=0x7f0a3c07a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.578+0000 7f0a42c00700 1 -- 172.21.15.44:0/1475471658 shutdown_connections 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.578+0000 7f0a42c00700 1 -- 172.21.15.44:0/1475471658 wait complete. 2024-04-16T11:30:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a42c00700 1 Processor -- start 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a42c00700 1 -- start start 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a42c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c117350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a42c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c117890 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c117350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c117350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55526/0 (socket says 172.21.15.44:55526) 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.579+0000 7f0a3be00700 1 -- 172.21.15.44:0/2193695845 learned_addr learned my addr 172.21.15.44:0/2193695845 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.580+0000 7f0a3be00700 1 -- 172.21.15.44:0/2193695845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c0094f0 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.580+0000 7f0a3be00700 1 --2- 172.21.15.44:0/2193695845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c117350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c003080 tx=0x7f0a2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.580+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a2c01c5c0 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.580+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c01c720 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.580+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3c117a90 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.580+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a2c01ca30 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.581+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3c118600 con 0x7f0a3c07ca40 2024-04-16T11:30:26.357 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.581+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0a2c01cc60 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.581+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0a2c01ac80 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.582+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3c05f5f0 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.585+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f0a2c01ae40 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.662+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0a3c10a400 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.670+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 7 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f0a2c01a450 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.670+0000 7f0a33e00700 1 -- 172.21.15.44:0/2193695845 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f0a3c10a400 con 0x7f0a3c07ca40 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.671+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 msgr2=0x7f0a3c117350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:26.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.671+0000 7f0a42c00700 1 --2- 172.21.15.44:0/2193695845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c117350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c003080 tx=0x7f0a2c016000 comp rx=0 tx=0).stop 2024-04-16T11:30:26.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.672+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 shutdown_connections 2024-04-16T11:30:26.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.672+0000 7f0a42c00700 1 --2- 172.21.15.44:0/2193695845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a3c07ca40 0x7f0a3c117350 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:26.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.672+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 >> 172.21.15.44:0/2193695845 conn(0x7f0a3c078350 msgr2=0x7f0a3c10db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:26.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.672+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 shutdown_connections 2024-04-16T11:30:26.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:25.672+0000 7f0a42c00700 1 -- 172.21.15.44:0/2193695845 wait complete. 2024-04-16T11:30:26.359 INFO:teuthology.orchestra.run.smithi044.stdout:Generating new minimal ceph.conf... 2024-04-16T11:30:27.459 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:27 smithi044 ceph-mon[28360]: from='client.? 172.21.15.44:0/665546942' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.009+0000 7f808d000700 1 Processor -- start 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.009+0000 7f808d000700 1 -- start start 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.009+0000 7f808d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80880772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.009+0000 7f808d000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088079120 con 0x7f8088078d10 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.010+0000 7f8085e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80880772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.010+0000 7f8085e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80880772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55540/0 (socket says 172.21.15.44:55540) 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.010+0000 7f8085e00700 1 -- 172.21.15.44:0/2400773341 learned_addr learned my addr 172.21.15.44:0/2400773341 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.010+0000 7f8085e00700 1 -- 172.21.15.44:0/2400773341 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8088077800 con 0x7f8088078d10 2024-04-16T11:30:27.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.011+0000 7f8085e00700 1 --2- 172.21.15.44:0/2400773341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80880772c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f80780098e0 tx=0x7f8078009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2cfa3f8713929 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.011+0000 7f8085400700 1 -- 172.21.15.44:0/2400773341 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f807800b860 con 0x7f8088078d10 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.011+0000 7f8085400700 1 -- 172.21.15.44:0/2400773341 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f807800be50 con 0x7f8088078d10 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.011+0000 7f8085400700 1 -- 172.21.15.44:0/2400773341 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8078010380 con 0x7f8088078d10 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.011+0000 7f808d000700 1 -- 172.21.15.44:0/2400773341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 msgr2=0x7f80880772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.011+0000 7f808d000700 1 --2- 172.21.15.44:0/2400773341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80880772c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f80780098e0 tx=0x7f8078009250 comp rx=0 tx=0).stop 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.012+0000 7f808d000700 1 -- 172.21.15.44:0/2400773341 shutdown_connections 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.012+0000 7f808d000700 1 --2- 172.21.15.44:0/2400773341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80880772c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.012+0000 7f808d000700 1 -- 172.21.15.44:0/2400773341 >> 172.21.15.44:0/2400773341 conn(0x7f80880fe090 msgr2=0x7f80881004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.012+0000 7f808d000700 1 -- 172.21.15.44:0/2400773341 shutdown_connections 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.012+0000 7f808d000700 1 -- 172.21.15.44:0/2400773341 wait complete. 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f808d000700 1 Processor -- start 2024-04-16T11:30:27.765 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f808d000700 1 -- start start 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f808d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80881137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f808d000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088113d20 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f8085e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80881137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f8085e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80881137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55542/0 (socket says 172.21.15.44:55542) 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.013+0000 7f8085e00700 1 -- 172.21.15.44:0/665546942 learned_addr learned my addr 172.21.15.44:0/665546942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.014+0000 7f8085e00700 1 -- 172.21.15.44:0/665546942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80780094f0 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.014+0000 7f8085e00700 1 --2- 172.21.15.44:0/665546942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80881137e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f807800ba90 tx=0x7f8078009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.014+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f807801a8a0 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.014+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f807801ae90 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.014+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8078019e80 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.014+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8088113f20 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.015+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8088114a90 con 0x7f8088078d10 2024-04-16T11:30:27.766 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.015+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f8078018460 con 0x7f8088078d10 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.015+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f80780237a0 con 0x7f8088078d10 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.016+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808805f5f0 con 0x7f8088078d10 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.019+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f8078028060 con 0x7f8088078d10 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.096+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8088108ac0 con 0x7f8088078d10 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.097+0000 7f807ea00700 1 -- 172.21.15.44:0/665546942 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f807801e070 con 0x7f8088078d10 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 msgr2=0x7f80881137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 --2- 172.21.15.44:0/665546942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80881137e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f807800ba90 tx=0x7f8078009250 comp rx=0 tx=0).stop 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 shutdown_connections 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 --2- 172.21.15.44:0/665546942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8088078d10 0x7f80881137e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 >> 172.21.15.44:0/665546942 conn(0x7f80880fe090 msgr2=0x7f80880feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 shutdown_connections 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:27.099+0000 7f808d000700 1 -- 172.21.15.44:0/665546942 wait complete. 2024-04-16T11:30:27.767 INFO:teuthology.orchestra.run.smithi044.stdout:Restarting the monitor... 2024-04-16T11:30:28.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:27 smithi044 systemd[1]: Stopping Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:30:28.630 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:28 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[28356]: 2024-04-16T11:30:28.272+0000 7f5942800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi044 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:30:28.630 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:28 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[28356]: 2024-04-16T11:30:28.272+0000 7f5942800700 -1 mon.smithi044@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T11:30:28.630 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:28 smithi044 podman[28605]: 2024-04-16 11:30:28.325647279 +0000 UTC m=+0.173161041 container died c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-16T11:30:28.924 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:28 smithi044 podman[28605]: 2024-04-16 11:30:28.732641148 +0000 UTC m=+0.580154935 container cleanup c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-16T11:30:28.924 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:28 smithi044 bash[28605]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044 2024-04-16T11:30:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:28 smithi044 podman[28619]: 2024-04-16 11:30:28.924959437 +0000 UTC m=+0.580072267 container remove c19cd9d3ef5a9b7954194d461cb556a17177a77f66894f3003fdeaa39ba0a9b6 (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T11:30:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:29 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service: Deactivated successfully. 2024-04-16T11:30:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:29 smithi044 systemd[1]: Stopped Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:30:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:29 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service: Consumed 1.961s CPU time. 2024-04-16T11:30:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:29 smithi044 systemd[1]: Starting Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:30:30.130 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:29 smithi044 podman[28711]: 2024-04-16 11:30:29.728385888 +0000 UTC m=+0.044243773 image pull b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 2024-04-16T11:30:30.130 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:29 smithi044 podman[28711]: 2024-04-16 11:30:29.881884497 +0000 UTC m=+0.197742412 container create 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T11:30:30.130 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 podman[28711]: 2024-04-16 11:30:30.131222599 +0000 UTC m=+0.447080500 container init 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-64502-quincy-3-45317f9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T11:30:30.214 INFO:teuthology.orchestra.run.smithi044.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 podman[28711]: 2024-04-16 11:30:30.1424416 +0000 UTC m=+0.458299505 container start 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, RELEASE=wip-64502-quincy-3-45317f9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-mon, pid 2 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: pidfile_write: ignore empty --pid-file 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: load: jerasure load: lrc 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: RocksDB version: 6.15.5 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Compile date Apr 12 2024 2024-04-16T11:30:30.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: DB SUMMARY 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: DB Session ID: SKZY7ICHDQ9C404KP95E 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: CURRENT file: CURRENT 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi044/store.db dir, Total Num: 1, files: 000008.sst 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi044/store.db: 000010.log size: 98282 ; 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.error_if_exists: 0 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.create_if_missing: 0 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.paranoid_checks: 1 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.env: 0x560939329b00 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.fs: Posix File System 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.info_log: 0x56096a92fc00 2024-04-16T11:30:30.480 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.statistics: (nil) 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.use_fsync: 0 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_log_file_size: 0 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.allow_fallocate: 1 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T11:30:30.481 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.use_direct_reads: 0 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.db_log_dir: 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi044/store.db 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T11:30:30.482 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.write_buffer_manager: 0x56096b6a4a20 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T11:30:30.483 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.unordered_write: 0 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.row_cache: None 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.wal_filter: None 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T11:30:30.484 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.preserve_deletes: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.two_write_queues: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.atomic_flush: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.log_readahead_size: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T11:30:30.485 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_background_jobs: 2 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_background_compactions: -1 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_subcompactions: 1 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_open_files: -1 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T11:30:30.486 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_background_flushes: -1 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Compression algorithms supported: 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kZSTD supported: 0 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kXpressCompression supported: 0 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kZlibCompression supported: 1 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kSnappyCompression supported: 1 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kLZ4Compression supported: 1 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: kBZip2Compression supported: 0 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi044/store.db/MANIFEST-000009 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.merge_operator: 2024-04-16T11:30:30.487 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_filter: None 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56096a984f58) 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: index_type: 0 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: data_block_index_type: 0 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: index_shortening: 1 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: hash_index_allow_collision: 1 2024-04-16T11:30:30.488 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: checksum: 1 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: no_block_cache: 0 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache: 0x56096a9b7090 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache_options: 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: capacity : 536870912 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: num_shard_bits : 4 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: strict_capacity_limit : 0 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache_compressed: (nil) 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: persistent_cache: (nil) 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_size: 4096 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_size_deviation: 10 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_restart_interval: 16 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: index_block_restart_interval: 1 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: metadata_block_size: 4096 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: partition_filters: 0 2024-04-16T11:30:30.489 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: use_delta_encoding: 1 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: whole_key_filtering: 1 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: verify_compression: 0 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: format_version: 4 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: enable_index_compression: 1 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_align: 0 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression: NoCompression 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.num_levels: 7 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T11:30:30.490 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T11:30:30.491 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T11:30:30.491 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T11:30:30.491 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T11:30:30.492 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T11:30:30.492 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T11:30:30.492 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T11:30:30.492 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T11:30:30.492 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T11:30:30.492 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T11:30:30.493 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.arena_block_size: 4194304 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T11:30:30.494 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.table_properties_collectors: 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.inplace_update_support: 0 2024-04-16T11:30:30.495 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.bloom_locality: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.max_successive_merges: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.ttl: 2592000 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.enable_blob_files: false 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.min_blob_size: 0 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T11:30:30.496 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi044/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267030198052, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267030199883, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93544, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91628, "index_size": 276, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84164, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713267030, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a93c19bf-c070-47fa-b590-3fb8fda71725", "db_session_id": "SKZY7ICHDQ9C404KP95E"}} 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267030200676, "job": 1, "event": "recovery_finished"} 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi044/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56096a94b180 2024-04-16T11:30:30.497 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: DB pointer 0x56096aa20000 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** DB Stats ** 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-16T11:30:30.498 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: L0 2/0 93.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Sum 2/0 93.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T11:30:30.499 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative compaction: 0.00 GB write, 12.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval compaction: 0.00 GB write, 12.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: L0 2/0 93.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Sum 2/0 93.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-16T11:30:30.500 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 70.1 0.00 0.00 1 0.001 0 0 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative compaction: 0.00 GB write, 12.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.501 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 bash[28711]: 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: starting mon.smithi044 rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi044 fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???) e1 preinit fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).mds e1 new map 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).mds e1 print_map 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: e1 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: legacy client fscid: -1 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: No filesystems configured 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T11:30:30.502 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 systemd[1]: Started Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mon.smithi044 is new leader, mons smithi044 in quorum (ranks 0) 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: monmap e1: 1 mons at {smithi044=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: fsmap 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T11:30:30.503 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:30 smithi044 ceph-mon[28731]: mgrmap e1: no daemons active 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.822+0000 7f545b000700 1 Processor -- start 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.823+0000 7f545b000700 1 -- start start 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.823+0000 7f545b000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f5454077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.823+0000 7f545b000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5454079070 con 0x7f5454078c60 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.824+0000 7f5453e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f5454077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.824+0000 7f5453e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f5454077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55550/0 (socket says 172.21.15.44:55550) 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.824+0000 7f5453e00700 1 -- 172.21.15.44:0/3564026583 learned_addr learned my addr 172.21.15.44:0/3564026583 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.824+0000 7f5453e00700 1 -- 172.21.15.44:0/3564026583 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540777a0 con 0x7f5454078c60 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.825+0000 7f5453e00700 1 --2- 172.21.15.44:0/3564026583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f5454077260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f54440098e0 tx=0x7f5444009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87aff17c41709891 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:31.529 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.825+0000 7f5453400700 1 -- 172.21.15.44:0/3564026583 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f544400b860 con 0x7f5454078c60 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.825+0000 7f5453400700 1 -- 172.21.15.44:0/3564026583 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f544400be50 con 0x7f5454078c60 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.825+0000 7f5453400700 1 -- 172.21.15.44:0/3564026583 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5444010380 con 0x7f5454078c60 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 -- 172.21.15.44:0/3564026583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 msgr2=0x7f5454077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 --2- 172.21.15.44:0/3564026583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f5454077260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f54440098e0 tx=0x7f5444009250 comp rx=0 tx=0).stop 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 -- 172.21.15.44:0/3564026583 shutdown_connections 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 --2- 172.21.15.44:0/3564026583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f5454077260 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 -- 172.21.15.44:0/3564026583 >> 172.21.15.44:0/3564026583 conn(0x7f54540fe0f0 msgr2=0x7f5454100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 -- 172.21.15.44:0/3564026583 shutdown_connections 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.826+0000 7f545b000700 1 -- 172.21.15.44:0/3564026583 wait complete. 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.827+0000 7f545b000700 1 Processor -- start 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.827+0000 7f545b000700 1 -- start start 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.828+0000 7f545b000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f54541137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.828+0000 7f545b000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5454113d20 con 0x7f5454078c60 2024-04-16T11:30:31.530 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.828+0000 7f5453e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f54541137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.828+0000 7f5453e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f54541137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55556/0 (socket says 172.21.15.44:55556) 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.828+0000 7f5453e00700 1 -- 172.21.15.44:0/2106474930 learned_addr learned my addr 172.21.15.44:0/2106474930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f5453e00700 1 -- 172.21.15.44:0/2106474930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440094f0 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f5453e00700 1 --2- 172.21.15.44:0/2106474930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f54541137e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5444006fd0 tx=0x7f5444009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f544401a7a0 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f544401ad90 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5444023e40 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5454113f20 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.829+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5454114a90 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.830+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f54440224b0 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.830+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f544401e070 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.831+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545405f5f0 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.834+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f5444023480 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.912+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f5454108b50 con 0x7f5454078c60 2024-04-16T11:30:31.531 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.917+0000 7f5450c00700 1 -- 172.21.15.44:0/2106474930 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5444023710 con 0x7f5454078c60 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.918+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 msgr2=0x7f54541137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.918+0000 7f545b000700 1 --2- 172.21.15.44:0/2106474930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f54541137e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5444006fd0 tx=0x7f5444009250 comp rx=0 tx=0).stop 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.919+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 shutdown_connections 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.919+0000 7f545b000700 1 --2- 172.21.15.44:0/2106474930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5454078c60 0x7f54541137e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.919+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 >> 172.21.15.44:0/2106474930 conn(0x7f54540fe0f0 msgr2=0x7f54540feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.919+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 shutdown_connections 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:30.919+0000 7f545b000700 1 -- 172.21.15.44:0/2106474930 wait complete. 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-16T11:30:31.532 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-16T11:30:31.533 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mgr... 2024-04-16T11:30:31.533 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 9283 ... 2024-04-16T11:30:32.091 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mgr.smithi044.sczobv 2024-04-16T11:30:32.092 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mgr.smithi044.sczobv.service: Unit ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mgr.smithi044.sczobv.service not loaded. 2024-04-16T11:30:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:31 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2106474930' entity='client.admin' 2024-04-16T11:30:32.656 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968.target.wants/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mgr.smithi044.sczobv.service → /etc/systemd/system/ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@.service. 2024-04-16T11:30:39.902 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T11:30:39.902 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-16T11:30:39.922 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T11:30:39.922 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-16T11:30:39.923 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-16T11:30:39.923 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T11:30:39.948 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T11:30:39.948 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-16T11:30:39.967 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T11:30:39.967 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-16T11:30:39.967 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-16T11:30:39.967 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-16T11:30:39.967 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr to start... 2024-04-16T11:30:39.967 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr... 2024-04-16T11:30:40.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:40 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3716185273' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T11:30:41.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:41.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:30:41.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "8b0307a8-fbe4-11ee-bc8f-c7b262605968", 2024-04-16T11:30:41.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T11:30:41.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T11:30:41.595 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T11:30:41.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T11:30:41.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T11:30:41.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T11:30:41.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-16T11:30:41.596 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "smithi044" 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T11:30:41.597 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T11:30:41.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T11:30:41.605 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:41.606 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T11:30:41.607 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T11:30:23.625575+0000", 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.659+0000 7f1fcaa00700 1 Processor -- start 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.659+0000 7f1fcaa00700 1 -- start start 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.659+0000 7f1fcaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4105040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.659+0000 7f1fcaa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4105610 con 0x7f1fc4104c30 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.660+0000 7f1fc3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.660+0000 7f1fc3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4105040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47998/0 (socket says 172.21.15.44:47998) 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.660+0000 7f1fc3e00700 1 -- 172.21.15.44:0/3689599193 learned_addr learned my addr 172.21.15.44:0/3689599193 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.661+0000 7f1fc3e00700 1 -- 172.21.15.44:0/3689599193 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4105750 con 0x7f1fc4104c30 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.661+0000 7f1fc3e00700 1 --2- 172.21.15.44:0/3689599193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4105040 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40098e0 tx=0x7f1fb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af3ca6ceb10a495 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:41.608 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.661+0000 7f1fc2a00700 1 -- 172.21.15.44:0/3689599193 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb400b860 con 0x7f1fc4104c30 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.661+0000 7f1fc2a00700 1 -- 172.21.15.44:0/3689599193 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f1fb400be50 con 0x7f1fc4104c30 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.661+0000 7f1fc2a00700 1 -- 172.21.15.44:0/3689599193 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4010380 con 0x7f1fc4104c30 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.662+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3689599193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 msgr2=0x7f1fc4105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.662+0000 7f1fcaa00700 1 --2- 172.21.15.44:0/3689599193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4105040 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40098e0 tx=0x7f1fb4009250 comp rx=0 tx=0).stop 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.662+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3689599193 shutdown_connections 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.662+0000 7f1fcaa00700 1 --2- 172.21.15.44:0/3689599193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4105040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.662+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3689599193 >> 172.21.15.44:0/3689599193 conn(0x7f1fc4100590 msgr2=0x7f1fc4102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.663+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3689599193 shutdown_connections 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.663+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3689599193 wait complete. 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.663+0000 7f1fcaa00700 1 Processor -- start 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.664+0000 7f1fcaa00700 1 -- start start 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.664+0000 7f1fcaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.664+0000 7f1fcaa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4113cc0 con 0x7f1fc4104c30 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.664+0000 7f1fc3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.664+0000 7f1fc3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48002/0 (socket says 172.21.15.44:48002) 2024-04-16T11:30:41.609 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.664+0000 7f1fc3e00700 1 -- 172.21.15.44:0/3716185273 learned_addr learned my addr 172.21.15.44:0/3716185273 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fc3e00700 1 -- 172.21.15.44:0/3716185273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40094f0 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fc3e00700 1 --2- 172.21.15.44:0/3716185273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4113780 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006fd0 tx=0x7f1fb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4010800 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f1fb4010df0 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb401ad80 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc4113ec0 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.665+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc4114a30 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.666+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1fb40243f0 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.666+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1fb4019710 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.667+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc405f5f0 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.671+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f1fb4029060 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.749+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1fc4106950 con 0x7f1fc4104c30 2024-04-16T11:30:41.610 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.749+0000 7f1fbbe00700 1 -- 172.21.15.44:0/3716185273 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f1fb401a460 con 0x7f1fc4104c30 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.750+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 msgr2=0x7f1fc4113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.750+0000 7f1fcaa00700 1 --2- 172.21.15.44:0/3716185273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4113780 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006fd0 tx=0x7f1fb4009250 comp rx=0 tx=0).stop 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.751+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 shutdown_connections 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.751+0000 7f1fcaa00700 1 --2- 172.21.15.44:0/3716185273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fc4104c30 0x7f1fc4113780 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.751+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 >> 172.21.15.44:0/3716185273 conn(0x7f1fc4100590 msgr2=0x7f1fc4101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.751+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 shutdown_connections 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:40.751+0000 7f1fcaa00700 1 -- 172.21.15.44:0/3716185273 wait complete. 2024-04-16T11:30:41.611 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (1/15)... 2024-04-16T11:30:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:44 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3321825773' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T11:30:45.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:45.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:30:45.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "8b0307a8-fbe4-11ee-bc8f-c7b262605968", 2024-04-16T11:30:45.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T11:30:45.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T11:30:45.395 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "smithi044" 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:45.396 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T11:30:45.397 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T11:30:45.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T11:30:45.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:45.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T11:30:23.625575+0000", 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.166+0000 7f79aec00700 1 Processor -- start 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.166+0000 7f79aec00700 1 -- start start 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.167+0000 7f79aec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.167+0000 7f79aec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8103250 con 0x7f79a8102900 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.167+0000 7f79a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.167+0000 7f79a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32906/0 (socket says 172.21.15.44:32906) 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.167+0000 7f79a7e00700 1 -- 172.21.15.44:0/3373181858 learned_addr learned my addr 172.21.15.44:0/3373181858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.168+0000 7f79a7e00700 1 -- 172.21.15.44:0/3373181858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a8103390 con 0x7f79a8102900 2024-04-16T11:30:45.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.168+0000 7f79a7e00700 1 --2- 172.21.15.44:0/3373181858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8102d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79980098e0 tx=0x7f7998009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936192a8884300ed server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.168+0000 7f79a6a00700 1 -- 172.21.15.44:0/3373181858 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f799800b860 con 0x7f79a8102900 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.168+0000 7f79a6a00700 1 -- 172.21.15.44:0/3373181858 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f799800be50 con 0x7f79a8102900 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.168+0000 7f79a6a00700 1 -- 172.21.15.44:0/3373181858 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7998010380 con 0x7f79a8102900 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.169+0000 7f79aec00700 1 -- 172.21.15.44:0/3373181858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 msgr2=0x7f79a8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.169+0000 7f79aec00700 1 --2- 172.21.15.44:0/3373181858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8102d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79980098e0 tx=0x7f7998009250 comp rx=0 tx=0).stop 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.169+0000 7f79aec00700 1 -- 172.21.15.44:0/3373181858 shutdown_connections 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.169+0000 7f79aec00700 1 --2- 172.21.15.44:0/3373181858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8102d10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.169+0000 7f79aec00700 1 -- 172.21.15.44:0/3373181858 >> 172.21.15.44:0/3373181858 conn(0x7f79a80fe0a0 msgr2=0x7f79a81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.169+0000 7f79aec00700 1 -- 172.21.15.44:0/3373181858 shutdown_connections 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.170+0000 7f79aec00700 1 -- 172.21.15.44:0/3373181858 wait complete. 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.171+0000 7f79aec00700 1 Processor -- start 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.171+0000 7f79aec00700 1 -- start start 2024-04-16T11:30:45.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.171+0000 7f79aec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8111610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.171+0000 7f79aec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8111b50 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.172+0000 7f79a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8111610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.172+0000 7f79a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8111610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32920/0 (socket says 172.21.15.44:32920) 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.172+0000 7f79a7e00700 1 -- 172.21.15.44:0/3321825773 learned_addr learned my addr 172.21.15.44:0/3321825773 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.172+0000 7f79a7e00700 1 -- 172.21.15.44:0/3321825773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980094f0 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.172+0000 7f79a7e00700 1 --2- 172.21.15.44:0/3321825773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8111610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7998000c00 tx=0x7f7998009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f799801a8a0 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f799801ae90 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a8111d50 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7998019e70 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a81128c0 con 0x7f79a8102900 2024-04-16T11:30:45.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f7998023450 con 0x7f79a8102900 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.173+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7998018770 con 0x7f79a8102900 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.174+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79880052e0 con 0x7f79a8102900 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.177+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f7998028060 con 0x7f79a8102900 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.255+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7988005860 con 0x7f79a8102900 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.256+0000 7f799fe00700 1 -- 172.21.15.44:0/3321825773 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f799801e070 con 0x7f79a8102900 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.257+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 msgr2=0x7f79a8111610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.257+0000 7f79aec00700 1 --2- 172.21.15.44:0/3321825773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8111610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7998000c00 tx=0x7f7998009250 comp rx=0 tx=0).stop 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.257+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 shutdown_connections 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.257+0000 7f79aec00700 1 --2- 172.21.15.44:0/3321825773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79a8102900 0x7f79a8111610 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.257+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 >> 172.21.15.44:0/3321825773 conn(0x7f79a80fe0a0 msgr2=0x7f79a80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.258+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 shutdown_connections 2024-04-16T11:30:45.415 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:44.258+0000 7f79aec00700 1 -- 172.21.15.44:0/3321825773 wait complete. 2024-04-16T11:30:45.416 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (2/15)... 2024-04-16T11:30:48.448 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:48 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3278630941' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T11:30:48.803 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:48.803 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:30:48.803 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "8b0307a8-fbe4-11ee-bc8f-c7b262605968", 2024-04-16T11:30:48.803 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T11:30:48.804 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T11:30:48.804 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T11:30:48.804 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T11:30:48.805 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "smithi044" 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.807 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T11:30:48.808 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.811 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.812 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T11:30:23.625575+0000", 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.989+0000 7f00b9800700 1 Processor -- start 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.989+0000 7f00b9800700 1 -- start start 2024-04-16T11:30:48.814 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.990+0000 7f00b9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.990+0000 7f00b9800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4103250 con 0x7f00b4102900 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.990+0000 7f00b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.990+0000 7f00b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32936/0 (socket says 172.21.15.44:32936) 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.990+0000 7f00b2a00700 1 -- 172.21.15.44:0/957001905 learned_addr learned my addr 172.21.15.44:0/957001905 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.991+0000 7f00b2a00700 1 -- 172.21.15.44:0/957001905 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4103390 con 0x7f00b4102900 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.991+0000 7f00b2a00700 1 --2- 172.21.15.44:0/957001905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4102d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f00a40098e0 tx=0x7f00a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6d1b94d6a9a518 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.991+0000 7f00b1600700 1 -- 172.21.15.44:0/957001905 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a400b860 con 0x7f00b4102900 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.991+0000 7f00b1600700 1 -- 172.21.15.44:0/957001905 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f00a400be50 con 0x7f00b4102900 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.991+0000 7f00b1600700 1 -- 172.21.15.44:0/957001905 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a4010380 con 0x7f00b4102900 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.992+0000 7f00b9800700 1 -- 172.21.15.44:0/957001905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 msgr2=0x7f00b4102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:48.815 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.992+0000 7f00b9800700 1 --2- 172.21.15.44:0/957001905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4102d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f00a40098e0 tx=0x7f00a4009250 comp rx=0 tx=0).stop 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.992+0000 7f00b9800700 1 -- 172.21.15.44:0/957001905 shutdown_connections 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.992+0000 7f00b9800700 1 --2- 172.21.15.44:0/957001905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4102d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.992+0000 7f00b9800700 1 -- 172.21.15.44:0/957001905 >> 172.21.15.44:0/957001905 conn(0x7f00b40fe0a0 msgr2=0x7f00b41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.992+0000 7f00b9800700 1 -- 172.21.15.44:0/957001905 shutdown_connections 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.993+0000 7f00b9800700 1 -- 172.21.15.44:0/957001905 wait complete. 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.993+0000 7f00b9800700 1 Processor -- start 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.993+0000 7f00b9800700 1 -- start start 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.994+0000 7f00b9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.994+0000 7f00b9800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40780a0 con 0x7f00b4102900 2024-04-16T11:30:48.819 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.994+0000 7f00b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.994+0000 7f00b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4077b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32942/0 (socket says 172.21.15.44:32942) 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.994+0000 7f00b2a00700 1 -- 172.21.15.44:0/3278630941 learned_addr learned my addr 172.21.15.44:0/3278630941 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.994+0000 7f00b2a00700 1 -- 172.21.15.44:0/3278630941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40094f0 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00b2a00700 1 --2- 172.21.15.44:0/3278630941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4077b60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f00a4000c00 tx=0x7f00a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a401a8a0 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f00a401ae90 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b4076110 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a4019e70 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b4076600 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f00a4018470 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.995+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f00a40237a0 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.996+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b404bed0 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:47.999+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f00a4028060 con 0x7f00b4102900 2024-04-16T11:30:48.820 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.077+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f00b4076a80 con 0x7f00b4102900 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.078+0000 7f00aaa00700 1 -- 172.21.15.44:0/3278630941 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f00a401e070 con 0x7f00b4102900 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.079+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 msgr2=0x7f00b4077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.079+0000 7f00b9800700 1 --2- 172.21.15.44:0/3278630941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4077b60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f00a4000c00 tx=0x7f00a4009250 comp rx=0 tx=0).stop 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.079+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 shutdown_connections 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.079+0000 7f00b9800700 1 --2- 172.21.15.44:0/3278630941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00b4102900 0x7f00b4077b60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.079+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 >> 172.21.15.44:0/3278630941 conn(0x7f00b40fe0a0 msgr2=0x7f00b40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.079+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 shutdown_connections 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:48.080+0000 7f00b9800700 1 -- 172.21.15.44:0/3278630941 wait complete. 2024-04-16T11:30:48.821 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (3/15)... 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: Activating manager daemon smithi044.sczobv 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: mgrmap e2: smithi044.sczobv(active, starting, since 0.0102876s) 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:30:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:30:49.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:30:49.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:30:49.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' 2024-04-16T11:30:49.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:30:49.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' 2024-04-16T11:30:49.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:49 smithi044 ceph-mon[28731]: from='mgr.14100 172.21.15.44:0/1308462692' entity='mgr.smithi044.sczobv' 2024-04-16T11:30:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:50 smithi044 ceph-mon[28731]: mgrmap e3: smithi044.sczobv(active, since 1.01898s) 2024-04-16T11:30:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:51 smithi044 ceph-mon[28731]: mgrmap e4: smithi044.sczobv(active, since 2s) 2024-04-16T11:30:52.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "8b0307a8-fbe4-11ee-bc8f-c7b262605968", 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "smithi044" 2024-04-16T11:30:52.421 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 21, 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T11:30:52.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T11:30:52.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T11:30:52.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T11:30:52.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T11:30:23.625575+0000", 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.409+0000 7f1b42200700 1 Processor -- start 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.409+0000 7f1b42200700 1 -- start start 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.409+0000 7f1b42200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c076190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.409+0000 7f1b42200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c077ff0 con 0x7f1b3c077be0 2024-04-16T11:30:52.434 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.410+0000 7f1b3b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.410+0000 7f1b3b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47274/0 (socket says 172.21.15.44:47274) 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.410+0000 7f1b3b400700 1 -- 172.21.15.44:0/2755756528 learned_addr learned my addr 172.21.15.44:0/2755756528 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.411+0000 7f1b3b400700 1 -- 172.21.15.44:0/2755756528 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0766d0 con 0x7f1b3c077be0 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.411+0000 7f1b3b400700 1 --2- 172.21.15.44:0/2755756528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c076190 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0098e0 tx=0x7f1b2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f38058f710ace20 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.411+0000 7f1b3a000700 1 -- 172.21.15.44:0/2755756528 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c00b860 con 0x7f1b3c077be0 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.411+0000 7f1b3a000700 1 -- 172.21.15.44:0/2755756528 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f1b2c00be50 con 0x7f1b3c077be0 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.411+0000 7f1b3a000700 1 -- 172.21.15.44:0/2755756528 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c010380 con 0x7f1b3c077be0 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 -- 172.21.15.44:0/2755756528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 msgr2=0x7f1b3c076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 --2- 172.21.15.44:0/2755756528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c076190 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0098e0 tx=0x7f1b2c009250 comp rx=0 tx=0).stop 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 -- 172.21.15.44:0/2755756528 shutdown_connections 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 --2- 172.21.15.44:0/2755756528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c076190 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 -- 172.21.15.44:0/2755756528 >> 172.21.15.44:0/2755756528 conn(0x7f1b3c0fe0a0 msgr2=0x7f1b3c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 -- 172.21.15.44:0/2755756528 shutdown_connections 2024-04-16T11:30:52.435 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.412+0000 7f1b42200700 1 -- 172.21.15.44:0/2755756528 wait complete. 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.413+0000 7f1b42200700 1 Processor -- start 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.413+0000 7f1b42200700 1 -- start start 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.413+0000 7f1b42200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.413+0000 7f1b42200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c113da0 con 0x7f1b3c077be0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.414+0000 7f1b3b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.414+0000 7f1b3b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47276/0 (socket says 172.21.15.44:47276) 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.414+0000 7f1b3b400700 1 -- 172.21.15.44:0/1496525898 learned_addr learned my addr 172.21.15.44:0/1496525898 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.414+0000 7f1b3b400700 1 -- 172.21.15.44:0/1496525898 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c0094f0 con 0x7f1b3c077be0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.414+0000 7f1b3b400700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c113860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c011cc0 tx=0x7f1b2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.415+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c010750 con 0x7f1b3c077be0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.415+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f1b2c010d40 con 0x7f1b3c077be0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.415+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c01ad50 con 0x7f1b3c077be0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.415+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c113fa0 con 0x7f1b3c077be0 2024-04-16T11:30:52.436 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.415+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c114b10 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.416+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f1b2c0108b0 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.416+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c05f5f0 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.416+0000 7f1b33400700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f1b200368a0 0x7f1b20038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.417+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1b2c04a380 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.417+0000 7f1b3aa00700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f1b200368a0 0x7f1b20038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.418+0000 7f1b3aa00700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f1b200368a0 0x7f1b20038d50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009990 tx=0x7f1b28006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.423+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1b2c018bd0 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.723+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1b3c04bed0 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.724+0000 7f1b33400700 1 -- 172.21.15.44:0/1496525898 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f1b2c020030 con 0x7f1b3c077be0 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.726+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f1b200368a0 msgr2=0x7f1b20038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.726+0000 7f1b42200700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f1b200368a0 0x7f1b20038d50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009990 tx=0x7f1b28006e40 comp rx=0 tx=0).stop 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.726+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 msgr2=0x7f1b3c113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.726+0000 7f1b42200700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c113860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c011cc0 tx=0x7f1b2c009250 comp rx=0 tx=0).stop 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.727+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 shutdown_connections 2024-04-16T11:30:52.437 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.727+0000 7f1b42200700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f1b200368a0 0x7f1b20038d50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:52.438 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.727+0000 7f1b42200700 1 --2- 172.21.15.44:0/1496525898 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b3c077be0 0x7f1b3c113860 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:52.438 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.727+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 >> 172.21.15.44:0/1496525898 conn(0x7f1b3c0fe0a0 msgr2=0x7f1b3c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:52.438 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.727+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 shutdown_connections 2024-04-16T11:30:52.438 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:51.727+0000 7f1b42200700 1 -- 172.21.15.44:0/1496525898 wait complete. 2024-04-16T11:30:52.438 INFO:teuthology.orchestra.run.smithi044.stdout:mgr is available 2024-04-16T11:30:52.661 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:52 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1496525898' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T11:30:53.919 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:53 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2153203968' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T11:30:53.932 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:53.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T11:30:53.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T11:30:53.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-16T11:30:53.935 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.038+0000 7f8836600700 1 Processor -- start 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.038+0000 7f8836600700 1 -- start start 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.039+0000 7f8836600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830078a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.039+0000 7f8836600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830079000 con 0x7f8830078620 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.039+0000 7f882f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830078a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:53.937 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.040+0000 7f882f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830078a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47278/0 (socket says 172.21.15.44:47278) 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.040+0000 7f882f400700 1 -- 172.21.15.44:0/3127704836 learned_addr learned my addr 172.21.15.44:0/3127704836 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.040+0000 7f882f400700 1 -- 172.21.15.44:0/3127704836 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8830079140 con 0x7f8830078620 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.040+0000 7f882f400700 1 --2- 172.21.15.44:0/3127704836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830078a30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f88200098e0 tx=0x7f8820009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d519bca844fe2d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.041+0000 7f882e000700 1 -- 172.21.15.44:0/3127704836 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882000b860 con 0x7f8830078620 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.041+0000 7f882e000700 1 -- 172.21.15.44:0/3127704836 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f882000be50 con 0x7f8830078620 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.041+0000 7f882e000700 1 -- 172.21.15.44:0/3127704836 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8820010380 con 0x7f8830078620 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.041+0000 7f8836600700 1 -- 172.21.15.44:0/3127704836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 msgr2=0x7f8830078a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.041+0000 7f8836600700 1 --2- 172.21.15.44:0/3127704836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830078a30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f88200098e0 tx=0x7f8820009250 comp rx=0 tx=0).stop 2024-04-16T11:30:53.938 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.042+0000 7f8836600700 1 -- 172.21.15.44:0/3127704836 shutdown_connections 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.042+0000 7f8836600700 1 --2- 172.21.15.44:0/3127704836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830078a30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.042+0000 7f8836600700 1 -- 172.21.15.44:0/3127704836 >> 172.21.15.44:0/3127704836 conn(0x7f8830100590 msgr2=0x7f8830102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.042+0000 7f8836600700 1 -- 172.21.15.44:0/3127704836 shutdown_connections 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.042+0000 7f8836600700 1 -- 172.21.15.44:0/3127704836 wait complete. 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f8836600700 1 Processor -- start 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f8836600700 1 -- start start 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f8836600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830117c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:53.942 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f8836600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830118140 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f882f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830117c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f882f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830117c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47288/0 (socket says 172.21.15.44:47288) 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.043+0000 7f882f400700 1 -- 172.21.15.44:0/2153203968 learned_addr learned my addr 172.21.15.44:0/2153203968 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.044+0000 7f882f400700 1 -- 172.21.15.44:0/2153203968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200094f0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.044+0000 7f882f400700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830117c00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.044+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882001a7d0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.044+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f882001adc0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.044+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830118340 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.045+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8820019ea0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.045+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8830118eb0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.046+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f8820019730 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.046+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883005f5f0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.046+0000 7f8827400700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f8814036850 0x7f8814038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.046+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8820049ff0 con 0x7f8830078620 2024-04-16T11:30:53.943 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.047+0000 7f882ea00700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f8814036850 0x7f8814038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.048+0000 7f882ea00700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f8814036850 0x7f8814038d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f881c009990 tx=0x7f881c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.051+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8820022d40 con 0x7f8830078620 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.266+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f883004bed0 con 0x7f8830078620 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.268+0000 7f8827400700 1 -- 172.21.15.44:0/2153203968 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f882001e030 con 0x7f8830078620 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f8814036850 msgr2=0x7f8814038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f8814036850 0x7f8814038d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f881c009990 tx=0x7f881c006e40 comp rx=0 tx=0).stop 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 msgr2=0x7f8830117c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830117c00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820009250 comp rx=0 tx=0).stop 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 shutdown_connections 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f8814036850 0x7f8814038d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 --2- 172.21.15.44:0/2153203968 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8830078620 0x7f8830117c00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 >> 172.21.15.44:0/2153203968 conn(0x7f8830100590 msgr2=0x7f8830102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.271+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 shutdown_connections 2024-04-16T11:30:53.944 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:53.272+0000 7f8836600700 1 -- 172.21.15.44:0/2153203968 wait complete. 2024-04-16T11:30:53.945 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling cephadm module... 2024-04-16T11:30:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:54 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1402175052' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-16T11:30:55.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.540+0000 7f552fc00700 1 Processor -- start 2024-04-16T11:30:55.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.540+0000 7f552fc00700 1 -- start start 2024-04-16T11:30:55.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.541+0000 7f552fc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f55280772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:55.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.541+0000 7f552fc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528079120 con 0x7f5528078d10 2024-04-16T11:30:55.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.541+0000 7f552d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f55280772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:55.598 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.542+0000 7f552d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f55280772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47298/0 (socket says 172.21.15.44:47298) 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.542+0000 7f552d200700 1 -- 172.21.15.44:0/1021929737 learned_addr learned my addr 172.21.15.44:0/1021929737 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.542+0000 7f552d200700 1 -- 172.21.15.44:0/1021929737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5528077800 con 0x7f5528078d10 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.542+0000 7f552d200700 1 --2- 172.21.15.44:0/1021929737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f55280772c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f551c0098e0 tx=0x7f551c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef3eab1e39ba0ea server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.543+0000 7f5527400700 1 -- 172.21.15.44:0/1021929737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f551c00b860 con 0x7f5528078d10 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.543+0000 7f5527400700 1 -- 172.21.15.44:0/1021929737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f551c00be50 con 0x7f5528078d10 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.543+0000 7f5527400700 1 -- 172.21.15.44:0/1021929737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f551c010380 con 0x7f5528078d10 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.543+0000 7f552fc00700 1 -- 172.21.15.44:0/1021929737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 msgr2=0x7f55280772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.543+0000 7f552fc00700 1 --2- 172.21.15.44:0/1021929737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f55280772c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f551c0098e0 tx=0x7f551c009250 comp rx=0 tx=0).stop 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.544+0000 7f552fc00700 1 -- 172.21.15.44:0/1021929737 shutdown_connections 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.544+0000 7f552fc00700 1 --2- 172.21.15.44:0/1021929737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f55280772c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.544+0000 7f552fc00700 1 -- 172.21.15.44:0/1021929737 >> 172.21.15.44:0/1021929737 conn(0x7f55280fe0e0 msgr2=0x7f55281004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.544+0000 7f552fc00700 1 -- 172.21.15.44:0/1021929737 shutdown_connections 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.544+0000 7f552fc00700 1 -- 172.21.15.44:0/1021929737 wait complete. 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552fc00700 1 Processor -- start 2024-04-16T11:30:55.599 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552fc00700 1 -- start start 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552fc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f5528117c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552fc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528118170 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f5528117c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f5528117c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47300/0 (socket says 172.21.15.44:47300) 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.545+0000 7f552d200700 1 -- 172.21.15.44:0/1402175052 learned_addr learned my addr 172.21.15.44:0/1402175052 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.546+0000 7f552d200700 1 -- 172.21.15.44:0/1402175052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c0094f0 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.546+0000 7f552d200700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f5528117c30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f551c006fd0 tx=0x7f551c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.546+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f551c01a7d0 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.546+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f551c01adc0 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.546+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5528118370 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.546+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f551c019ea0 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.547+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5528118ee0 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.548+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552805f5f0 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.548+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f551c019730 con 0x7f5528078d10 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.548+0000 7f5524c00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f5510036850 0x7f5510038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:55.600 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.548+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f551c049ff0 con 0x7f5528078d10 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.549+0000 7f5527e00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f5510036850 0x7f5510038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.549+0000 7f5527e00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f5510036850 0x7f5510038d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5514009990 tx=0x7f5514006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.554+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f551c022b60 con 0x7f5528078d10 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.828+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f552804bed0 con 0x7f5528078d10 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.886+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f551c01fd40 con 0x7f5528078d10 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.887+0000 7f5524c00700 1 -- 172.21.15.44:0/1402175052 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f551c04e450 con 0x7f5528078d10 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.890+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f5510036850 msgr2=0x7f5510038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.890+0000 7f552fc00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f5510036850 0x7f5510038d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5514009990 tx=0x7f5514006e40 comp rx=0 tx=0).stop 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.890+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 msgr2=0x7f5528117c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.890+0000 7f552fc00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f5528117c30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f551c006fd0 tx=0x7f551c009250 comp rx=0 tx=0).stop 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.891+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 shutdown_connections 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.891+0000 7f552fc00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f5510036850 0x7f5510038d00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.891+0000 7f552fc00700 1 --2- 172.21.15.44:0/1402175052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5528078d10 0x7f5528117c30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:55.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.891+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 >> 172.21.15.44:0/1402175052 conn(0x7f55280fe0e0 msgr2=0x7f552810f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:55.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.891+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 shutdown_connections 2024-04-16T11:30:55.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:54.891+0000 7f552fc00700 1 -- 172.21.15.44:0/1402175052 wait complete. 2024-04-16T11:30:56.162 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:55 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1402175052' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-16T11:30:56.162 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:55 smithi044 ceph-mon[28731]: mgrmap e5: smithi044.sczobv(active, since 6s) 2024-04-16T11:30:57.046 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:30:56 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/679105634' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T11:30:57.215 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:30:57.215 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-16T11:30:57.215 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "smithi044.sczobv", 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.225+0000 7fbb8b000700 1 Processor -- start 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.225+0000 7fbb8b000700 1 -- start start 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.226+0000 7fbb8b000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84076190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.226+0000 7fbb8b000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb84077ff0 con 0x7fbb84077be0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.226+0000 7fbb83e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.226+0000 7fbb83e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47332/0 (socket says 172.21.15.44:47332) 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.226+0000 7fbb83e00700 1 -- 172.21.15.44:0/3745861495 learned_addr learned my addr 172.21.15.44:0/3745861495 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.227+0000 7fbb83e00700 1 -- 172.21.15.44:0/3745861495 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb840766d0 con 0x7fbb84077be0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.227+0000 7fbb83e00700 1 --2- 172.21.15.44:0/3745861495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84076190 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb740098e0 tx=0x7fbb74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c1b2963e610a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.227+0000 7fbb82a00700 1 -- 172.21.15.44:0/3745861495 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb7400b860 con 0x7fbb84077be0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.227+0000 7fbb82a00700 1 -- 172.21.15.44:0/3745861495 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fbb7400be50 con 0x7fbb84077be0 2024-04-16T11:30:57.216 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.227+0000 7fbb82a00700 1 -- 172.21.15.44:0/3745861495 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb74010380 con 0x7fbb84077be0 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 -- 172.21.15.44:0/3745861495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 msgr2=0x7fbb84076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 --2- 172.21.15.44:0/3745861495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84076190 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb740098e0 tx=0x7fbb74009250 comp rx=0 tx=0).stop 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 -- 172.21.15.44:0/3745861495 shutdown_connections 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 --2- 172.21.15.44:0/3745861495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84076190 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 -- 172.21.15.44:0/3745861495 >> 172.21.15.44:0/3745861495 conn(0x7fbb840fe0a0 msgr2=0x7fbb841004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 -- 172.21.15.44:0/3745861495 shutdown_connections 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.228+0000 7fbb8b000700 1 -- 172.21.15.44:0/3745861495 wait complete. 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.229+0000 7fbb8b000700 1 Processor -- start 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.229+0000 7fbb8b000700 1 -- start start 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.230+0000 7fbb8b000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.230+0000 7fbb8b000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb84113da0 con 0x7fbb84077be0 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.230+0000 7fbb83e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.230+0000 7fbb83e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47334/0 (socket says 172.21.15.44:47334) 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.230+0000 7fbb83e00700 1 -- 172.21.15.44:0/679105634 learned_addr learned my addr 172.21.15.44:0/679105634 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.230+0000 7fbb83e00700 1 -- 172.21.15.44:0/679105634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740094f0 con 0x7fbb84077be0 2024-04-16T11:30:57.217 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.231+0000 7fbb83e00700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84113860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbb74006fd0 tx=0x7fbb74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.231+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb7401a7d0 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.231+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fbb7401adc0 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.231+0000 7fbb8b000700 1 -- 172.21.15.44:0/679105634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb84113fa0 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.231+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb74019ea0 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.231+0000 7fbb8b000700 1 -- 172.21.15.44:0/679105634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb84114b10 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.232+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fbb74021440 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.232+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5c0052b0 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.233+0000 7fbb7be00700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 0x7fbb68038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.233+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbb7404a270 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.233+0000 7fbb83400700 1 -- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 msgr2=0x7fbb68038d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.233+0000 7fbb83400700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 0x7fbb68038d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.238+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbb7402a710 con 0x7fbb84077be0 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.434+0000 7fbb83400700 1 -- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 msgr2=0x7fbb68038d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:30:57.218 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.434+0000 7fbb83400700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 0x7fbb68038d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.510+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbb5c005d70 con 0x7fbb84077be0 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.510+0000 7fbb7be00700 1 -- 172.21.15.44:0/679105634 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fbb7401e070 con 0x7fbb84077be0 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 msgr2=0x7fbb68038d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 0x7fbb68038d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 msgr2=0x7fbb84113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84113860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbb74006fd0 tx=0x7fbb74009250 comp rx=0 tx=0).stop 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 shutdown_connections 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7fbb68036850 0x7fbb68038d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 --2- 172.21.15.44:0/679105634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb84077be0 0x7fbb84113860 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 >> 172.21.15.44:0/679105634 conn(0x7fbb840fe0a0 msgr2=0x7fbb840feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 shutdown_connections 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:56.513+0000 7fbb79600700 1 -- 172.21.15.44:0/679105634 wait complete. 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-04-16T11:30:57.219 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 5... 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: Active manager daemon smithi044.sczobv restarted 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: Activating manager daemon smithi044.sczobv 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: osdmap e2: 0 total, 0 up, 0 in 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: mgrmap e6: smithi044.sczobv(active, starting, since 0.00875036s) 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:31:03.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:03 smithi044 ceph-mon[28731]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: Found migration_current of "None". Setting to last migration. 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:31:04.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:04 smithi044 ceph-mon[28731]: mgrmap e7: smithi044.sczobv(active, since 1.01646s) 2024-04-16T11:31:05.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:31:05.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-16T11:31:05.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T11:31:05.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:31:05.414 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.757+0000 7f95d5c00700 1 Processor -- start 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.758+0000 7f95d5c00700 1 -- start start 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.758+0000 7f95d5c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.758+0000 7f95d5c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0105a40 con 0x7f95d00712d0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.758+0000 7f95cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.759+0000 7f95cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47342/0 (socket says 172.21.15.44:47342) 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.759+0000 7f95cea00700 1 -- 172.21.15.44:0/3363806364 learned_addr learned my addr 172.21.15.44:0/3363806364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.759+0000 7f95cea00700 1 -- 172.21.15.44:0/3363806364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d0105b80 con 0x7f95d00712d0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.759+0000 7f95cea00700 1 --2- 172.21.15.44:0/3363806364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0105500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f95c00098e0 tx=0x7f95c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e32c697764eb7fc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.760+0000 7f95cd600700 1 -- 172.21.15.44:0/3363806364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95c000b860 con 0x7f95d00712d0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.760+0000 7f95cd600700 1 -- 172.21.15.44:0/3363806364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f95c000be50 con 0x7f95d00712d0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.760+0000 7f95cd600700 1 -- 172.21.15.44:0/3363806364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95c0010380 con 0x7f95d00712d0 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.760+0000 7f95d5c00700 1 -- 172.21.15.44:0/3363806364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 msgr2=0x7f95d0105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.760+0000 7f95d5c00700 1 --2- 172.21.15.44:0/3363806364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0105500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f95c00098e0 tx=0x7f95c0009250 comp rx=0 tx=0).stop 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.761+0000 7f95d5c00700 1 -- 172.21.15.44:0/3363806364 shutdown_connections 2024-04-16T11:31:05.416 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.761+0000 7f95d5c00700 1 --2- 172.21.15.44:0/3363806364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0105500 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.761+0000 7f95d5c00700 1 -- 172.21.15.44:0/3363806364 >> 172.21.15.44:0/3363806364 conn(0x7f95d00fcfd0 msgr2=0x7f95d00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.761+0000 7f95d5c00700 1 -- 172.21.15.44:0/3363806364 shutdown_connections 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.761+0000 7f95d5c00700 1 -- 172.21.15.44:0/3363806364 wait complete. 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.762+0000 7f95d5c00700 1 Processor -- start 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.762+0000 7f95d5c00700 1 -- start start 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.762+0000 7f95d5c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0102e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.763+0000 7f95d5c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0103390 con 0x7f95d00712d0 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.763+0000 7f95cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.763+0000 7f95cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0102e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47344/0 (socket says 172.21.15.44:47344) 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.763+0000 7f95cea00700 1 -- 172.21.15.44:0/392315130 learned_addr learned my addr 172.21.15.44:0/392315130 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:05.419 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.763+0000 7f95cea00700 1 -- 172.21.15.44:0/392315130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c00094f0 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.763+0000 7f95cea00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0102e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f95c0006fd0 tx=0x7f95c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.764+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95c001a7d0 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.764+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f95c001adc0 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.764+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95c0019ea0 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.764+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d0101400 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.764+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d01018f0 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.765+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f95c0019730 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.765+0000 7f95c6a00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.765+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 --> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95b4039460 con 0x7f95b40368a0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.765+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f95c004a000 con 0x7f95d00712d0 2024-04-16T11:31:05.420 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.766+0000 7f95ce000700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 msgr2=0x7f95b4038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:31:05.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.766+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:31:05.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.967+0000 7f95ce000700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 msgr2=0x7f95b4038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:31:05.423 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:57.967+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:58.368+0000 7f95ce000700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 msgr2=0x7f95b4038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:58.368+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:59.169+0000 7f95ce000700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 msgr2=0x7f95b4038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:30:59.169+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:00.771+0000 7f95ce000700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 msgr2=0x7f95b4038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/627376257 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:00.771+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:03.696+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 6) v1 ==== 43143+0+0 (secure 0 0 0) 0x7f95c001fd40 con 0x7f95d00712d0 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:03.696+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 msgr2=0x7f95b4038d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:03.696+0000 7f95c6a00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.700+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 7) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f95c00223b0 con 0x7f95d00712d0 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.701+0000 7f95c6a00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f95b4039ff0 0x7f95b403c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.701+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95b4039460 con 0x7f95b4039ff0 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.703+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f95b4039ff0 0x7f95b403c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:05.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.706+0000 7f95ce000700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f95b4039ff0 0x7f95b403c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0039d0 tx=0x7f95bc009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.708+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f95b4039460 con 0x7f95b4039ff0 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.715+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f95d004bed0 con 0x7f95b4039ff0 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.715+0000 7f95c6a00700 1 -- 172.21.15.44:0/392315130 <== mgr.14120 v2:172.21.15.44:6800/2358899774 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f95d004bed0 con 0x7f95b4039ff0 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f95b4039ff0 msgr2=0x7f95b403c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f95b4039ff0 0x7f95b403c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0039d0 tx=0x7f95bc009350 comp rx=0 tx=0).stop 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 msgr2=0x7f95d0102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0102e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f95c0006fd0 tx=0x7f95c0009250 comp rx=0 tx=0).stop 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 shutdown_connections 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f95b4039ff0 0x7f95b403c3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f95d00712d0 0x7f95d0102e50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 --2- 172.21.15.44:0/392315130 >> [v2:172.21.15.44:6800/627376257,v1:172.21.15.44:6801/627376257] conn(0x7f95b40368a0 0x7f95b4038d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 >> 172.21.15.44:0/392315130 conn(0x7f95d00fcfd0 msgr2=0x7f95d00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 shutdown_connections 2024-04-16T11:31:05.425 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:04.716+0000 7f95d5c00700 1 -- 172.21.15.44:0/392315130 wait complete. 2024-04-16T11:31:05.426 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 5 is available 2024-04-16T11:31:05.426 INFO:teuthology.orchestra.run.smithi044.stdout:Setting orchestrator backend to cephadm... 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: [16/Apr/2024:11:31:04] ENGINE Bus STARTING 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: [16/Apr/2024:11:31:04] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: [16/Apr/2024:11:31:04] ENGINE Bus STARTED 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:05.993 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:05 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:06.971 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:06 smithi044 ceph-mon[28731]: mgrmap e8: smithi044.sczobv(active, since 2s) 2024-04-16T11:31:06.971 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:06 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:06.971 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:06 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:06.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.070+0000 7f7e23a00700 1 Processor -- start 2024-04-16T11:31:06.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.070+0000 7f7e23a00700 1 -- start start 2024-04-16T11:31:06.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.071+0000 7f7e23a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c105050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:06.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.071+0000 7f7e23a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c105620 con 0x7f7e1c104c20 2024-04-16T11:31:06.983 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.071+0000 7f7e21000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.071+0000 7f7e21000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53702/0 (socket says 172.21.15.44:53702) 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.072+0000 7f7e21000700 1 -- 172.21.15.44:0/147866196 learned_addr learned my addr 172.21.15.44:0/147866196 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.072+0000 7f7e21000700 1 -- 172.21.15.44:0/147866196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c105760 con 0x7f7e1c104c20 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.072+0000 7f7e21000700 1 --2- 172.21.15.44:0/147866196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c105050 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e100098e0 tx=0x7f7e10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cbd9443f1b303c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.073+0000 7f7e1b400700 1 -- 172.21.15.44:0/147866196 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e1000b860 con 0x7f7e1c104c20 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.073+0000 7f7e1b400700 1 -- 172.21.15.44:0/147866196 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f7e1000be50 con 0x7f7e1c104c20 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.073+0000 7f7e1b400700 1 -- 172.21.15.44:0/147866196 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e10010380 con 0x7f7e1c104c20 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 -- 172.21.15.44:0/147866196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 msgr2=0x7f7e1c105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 --2- 172.21.15.44:0/147866196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c105050 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e100098e0 tx=0x7f7e10009250 comp rx=0 tx=0).stop 2024-04-16T11:31:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 -- 172.21.15.44:0/147866196 shutdown_connections 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 --2- 172.21.15.44:0/147866196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c105050 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 -- 172.21.15.44:0/147866196 >> 172.21.15.44:0/147866196 conn(0x7f7e1c078560 msgr2=0x7f7e1c078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 -- 172.21.15.44:0/147866196 shutdown_connections 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.074+0000 7f7e23a00700 1 -- 172.21.15.44:0/147866196 wait complete. 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.075+0000 7f7e23a00700 1 Processor -- start 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.075+0000 7f7e23a00700 1 -- start start 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.075+0000 7f7e23a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c1137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.075+0000 7f7e23a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c113cf0 con 0x7f7e1c104c20 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.076+0000 7f7e21000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c1137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.076+0000 7f7e21000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c1137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53710/0 (socket says 172.21.15.44:53710) 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.076+0000 7f7e21000700 1 -- 172.21.15.44:0/197266613 learned_addr learned my addr 172.21.15.44:0/197266613 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.076+0000 7f7e21000700 1 -- 172.21.15.44:0/197266613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100094f0 con 0x7f7e1c104c20 2024-04-16T11:31:06.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.076+0000 7f7e21000700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c1137b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000ba90 tx=0x7f7e10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.077+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e1001a8a0 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.077+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f7e1001ae90 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.077+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c113ef0 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.077+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e10019ea0 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.077+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c114a60 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.078+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f7e10019750 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.078+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c05f5f0 con 0x7f7e1c104c20 2024-04-16T11:31:06.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.078+0000 7f7e18c00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7e040368a0 0x7f7e04038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.078+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7e10049ff0 con 0x7f7e1c104c20 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.079+0000 7f7e1be00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7e040368a0 0x7f7e04038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.079+0000 7f7e1be00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7e040368a0 0x7f7e04038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009990 tx=0x7f7e08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.084+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7e10022ba0 con 0x7f7e1c104c20 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.315+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e1c105760 con 0x7f7e040368a0 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.337+0000 7f7e18c00700 1 -- 172.21.15.44:0/197266613 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e1c105760 con 0x7f7e040368a0 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7e040368a0 msgr2=0x7f7e04038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7e040368a0 0x7f7e04038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009990 tx=0x7f7e08006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 msgr2=0x7f7e1c1137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c1137b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000ba90 tx=0x7f7e10009250 comp rx=0 tx=0).stop 2024-04-16T11:31:06.990 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 shutdown_connections 2024-04-16T11:31:06.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7e040368a0 0x7f7e04038d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:06.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 --2- 172.21.15.44:0/197266613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7e1c104c20 0x7f7e1c1137b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:06.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.340+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 >> 172.21.15.44:0/197266613 conn(0x7f7e1c078560 msgr2=0x7f7e1c102430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:06.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.341+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 shutdown_connections 2024-04-16T11:31:06.991 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:06.341+0000 7f7e23a00700 1 -- 172.21.15.44:0/197266613 wait complete. 2024-04-16T11:31:07.859 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:07 smithi044 ceph-mon[28731]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.607+0000 7f20f3200700 1 Processor -- start 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.607+0000 7f20f3200700 1 -- start start 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.608+0000 7f20f3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec07ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.608+0000 7f20f3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec07d420 con 0x7f20ec07ca40 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.608+0000 7f20ebe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec07ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:08.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.608+0000 7f20ebe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec07ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53724/0 (socket says 172.21.15.44:53724) 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.608+0000 7f20ebe00700 1 -- 172.21.15.44:0/681223924 learned_addr learned my addr 172.21.15.44:0/681223924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.609+0000 7f20ebe00700 1 -- 172.21.15.44:0/681223924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec07d560 con 0x7f20ec07ca40 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.609+0000 7f20ebe00700 1 --2- 172.21.15.44:0/681223924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec07ce50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0098e0 tx=0x7f20dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f692171a0a6711 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.609+0000 7f20eaa00700 1 -- 172.21.15.44:0/681223924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20dc00b860 con 0x7f20ec07ca40 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.609+0000 7f20eaa00700 1 -- 172.21.15.44:0/681223924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f20dc00be50 con 0x7f20ec07ca40 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.609+0000 7f20eaa00700 1 -- 172.21.15.44:0/681223924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20dc010380 con 0x7f20ec07ca40 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 -- 172.21.15.44:0/681223924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 msgr2=0x7f20ec07ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 --2- 172.21.15.44:0/681223924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec07ce50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0098e0 tx=0x7f20dc009250 comp rx=0 tx=0).stop 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 -- 172.21.15.44:0/681223924 shutdown_connections 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 --2- 172.21.15.44:0/681223924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec07ce50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 -- 172.21.15.44:0/681223924 >> 172.21.15.44:0/681223924 conn(0x7f20ec078350 msgr2=0x7f20ec07a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 -- 172.21.15.44:0/681223924 shutdown_connections 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.610+0000 7f20f3200700 1 -- 172.21.15.44:0/681223924 wait complete. 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.611+0000 7f20f3200700 1 Processor -- start 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.611+0000 7f20f3200700 1 -- start start 2024-04-16T11:31:08.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.612+0000 7f20f3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.612+0000 7f20f3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec118120 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.612+0000 7f20ebe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.612+0000 7f20ebe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53738/0 (socket says 172.21.15.44:53738) 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.612+0000 7f20ebe00700 1 -- 172.21.15.44:0/2410729581 learned_addr learned my addr 172.21.15.44:0/2410729581 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.612+0000 7f20ebe00700 1 -- 172.21.15.44:0/2410729581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc0094f0 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.613+0000 7f20ebe00700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec117be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f20dc000c00 tx=0x7f20dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.613+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20dc01a8a0 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.613+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f20dc01ae90 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.613+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20dc019e70 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.613+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec118320 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.613+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec118e90 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.614+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f20dc019730 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.614+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec05f5f0 con 0x7f20ec07ca40 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.615+0000 7f20e3e00700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f20d00368a0 0x7f20d0038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:08.512 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.615+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f20dc04a190 con 0x7f20ec07ca40 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.615+0000 7f20eb400700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f20d00368a0 0x7f20d0038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.616+0000 7f20eb400700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f20d00368a0 0x7f20d0038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009990 tx=0x7f20d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.620+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f20dc021b50 con 0x7f20ec07ca40 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.840+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f20ec07a4d0 con 0x7f20d00368a0 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.842+0000 7f20e3e00700 1 -- 172.21.15.44:0/2410729581 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f20ec07a4d0 con 0x7f20d00368a0 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f20d00368a0 msgr2=0x7f20d0038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f20d00368a0 0x7f20d0038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009990 tx=0x7f20d8006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 msgr2=0x7f20ec117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec117be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f20dc000c00 tx=0x7f20dc009250 comp rx=0 tx=0).stop 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 shutdown_connections 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f20d00368a0 0x7f20d0038d50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 --2- 172.21.15.44:0/2410729581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec07ca40 0x7f20ec117be0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 >> 172.21.15.44:0/2410729581 conn(0x7f20ec078350 msgr2=0x7f20ec079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 shutdown_connections 2024-04-16T11:31:08.513 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:07.845+0000 7f20f3200700 1 -- 172.21.15.44:0/2410729581 wait complete. 2024-04-16T11:31:08.514 INFO:teuthology.orchestra.run.smithi044.stdout:Generating ssh key... 2024-04-16T11:31:09.135 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:08 smithi044 ceph-mon[28731]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:10.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.137+0000 7f311dc00700 1 Processor -- start 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.138+0000 7f311dc00700 1 -- start start 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.138+0000 7f311dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f3118107720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.138+0000 7f311dc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3118103110 con 0x7f3118071300 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.139+0000 7f3117e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f3118107720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.139+0000 7f3117e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f3118107720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53742/0 (socket says 172.21.15.44:53742) 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.139+0000 7f3117e00700 1 -- 172.21.15.44:0/3908363450 learned_addr learned my addr 172.21.15.44:0/3908363450 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:10.232 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.139+0000 7f3117e00700 1 -- 172.21.15.44:0/3908363450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3118107c60 con 0x7f3118071300 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.140+0000 7f3117e00700 1 --2- 172.21.15.44:0/3908363450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f3118107720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f31080098e0 tx=0x7f3108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d277301354114883 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.140+0000 7f3116a00700 1 -- 172.21.15.44:0/3908363450 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310800b860 con 0x7f3118071300 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.140+0000 7f3116a00700 1 -- 172.21.15.44:0/3908363450 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f310800be50 con 0x7f3118071300 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.140+0000 7f3116a00700 1 -- 172.21.15.44:0/3908363450 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3108010380 con 0x7f3118071300 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 -- 172.21.15.44:0/3908363450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 msgr2=0x7f3118107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 --2- 172.21.15.44:0/3908363450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f3118107720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f31080098e0 tx=0x7f3108009250 comp rx=0 tx=0).stop 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 -- 172.21.15.44:0/3908363450 shutdown_connections 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 --2- 172.21.15.44:0/3908363450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f3118107720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 -- 172.21.15.44:0/3908363450 >> 172.21.15.44:0/3908363450 conn(0x7f31180fcd30 msgr2=0x7f31180ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 -- 172.21.15.44:0/3908363450 shutdown_connections 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.141+0000 7f311dc00700 1 -- 172.21.15.44:0/3908363450 wait complete. 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.142+0000 7f311dc00700 1 Processor -- start 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.142+0000 7f311dc00700 1 -- start start 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.142+0000 7f311dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f31181136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:10.233 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.142+0000 7f311dc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3118113bf0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.143+0000 7f3117e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f31181136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.143+0000 7f3117e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f31181136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53750/0 (socket says 172.21.15.44:53750) 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.143+0000 7f3117e00700 1 -- 172.21.15.44:0/1430198165 learned_addr learned my addr 172.21.15.44:0/1430198165 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.143+0000 7f3117e00700 1 -- 172.21.15.44:0/1430198165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31080094f0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.143+0000 7f3117e00700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f31181136b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3108006fd0 tx=0x7f3108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.143+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310801a7d0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.144+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f310801adc0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.144+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3118113df0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.144+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3108019ea0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.144+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3118114960 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.145+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f3108019730 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.145+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311805f5f0 con 0x7f3118071300 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.145+0000 7f310fe00700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f30fc0368a0 0x7f30fc038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:10.234 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.145+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f310804a0d0 con 0x7f3118071300 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.146+0000 7f3117400700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f30fc0368a0 0x7f30fc038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.146+0000 7f3117400700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f30fc0368a0 0x7f30fc038d50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3104009990 tx=0x7f3104006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.151+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f310807c250 con 0x7f3118071300 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.367+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f31180fdf20 con 0x7f30fc0368a0 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.517+0000 7f310fe00700 1 -- 172.21.15.44:0/1430198165 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31180fdf20 con 0x7f30fc0368a0 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f30fc0368a0 msgr2=0x7f30fc038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f30fc0368a0 0x7f30fc038d50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3104009990 tx=0x7f3104006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 msgr2=0x7f31181136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f31181136b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3108006fd0 tx=0x7f3108009250 comp rx=0 tx=0).stop 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 shutdown_connections 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f30fc0368a0 0x7f30fc038d50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 --2- 172.21.15.44:0/1430198165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3118071300 0x7f31181136b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 >> 172.21.15.44:0/1430198165 conn(0x7f31180fcd30 msgr2=0x7f31180fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:10.235 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 shutdown_connections 2024-04-16T11:31:10.236 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:09.521+0000 7f311dc00700 1 -- 172.21.15.44:0/1430198165 wait complete. 2024-04-16T11:31:10.888 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:10 smithi044 ceph-mon[28731]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:10.888 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:10 smithi044 ceph-mon[28731]: Generating ssh key... 2024-04-16T11:31:10.888 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:10 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:10.888 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:10 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:11.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+RBEvnKbLvMm+wSwSX90n8BwDW4FZEw4F8db7FtqTsjzS1+L5Lv82c4nXK6d9tDhtVoaqCJNVlZ4HRYCATAqPnXeGd5Rxi091osSoF40Cu0iKbG7aajiVCpNF4/hzV6PNAZNsauDalianHQGD7gsvC9Cs69IvyocSVKV1u9mmROjsZl8vG/WmLv9I6pRmN/iq31McYdXEj0vu4Ch91y+dt0BqhyaLJJT8pJCv055uSI7D1r84Czqdft7szLYJNbpkZmNRQ+5pD68SJ29sprc5wS153N500CsiLZLwvBpNBlxOh1OHIIM/N+GIjnpzFUNYFw9XHHg35FgAH1Sn+lEWOYIh5zPeS2WzcQ3rBxJ7O09WGCWG+h2d07T2O4DFxQMmSbubVQrkxTKKfrBktbPATqbfPRJzrhjrDw+npQ7biMEGNSt4Gvmm8lJvnarr919FJFRd+gN066YVIoZNeQ5IVphg8soBJAMYSMxIsMeituBeX39JsotaBDlVpxp+UuM= ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:31:11.978 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.044+0000 7fb8dea00700 1 Processor -- start 2024-04-16T11:31:11.978 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.045+0000 7fb8dea00700 1 -- start start 2024-04-16T11:31:11.978 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.045+0000 7fb8dea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8076130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.045+0000 7fb8dea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d8077f90 con 0x7fb8d8077b80 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.045+0000 7fb8d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8076130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.046+0000 7fb8d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8076130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53758/0 (socket says 172.21.15.44:53758) 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.046+0000 7fb8d7e00700 1 -- 172.21.15.44:0/3511248639 learned_addr learned my addr 172.21.15.44:0/3511248639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.046+0000 7fb8d7e00700 1 -- 172.21.15.44:0/3511248639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8076670 con 0x7fb8d8077b80 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.046+0000 7fb8d7e00700 1 --2- 172.21.15.44:0/3511248639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8076130 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80098e0 tx=0x7fb8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc7ddfda25fe818 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:11.979 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.047+0000 7fb8d6a00700 1 -- 172.21.15.44:0/3511248639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c800b860 con 0x7fb8d8077b80 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.047+0000 7fb8d6a00700 1 -- 172.21.15.44:0/3511248639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb8c800be50 con 0x7fb8d8077b80 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.047+0000 7fb8d6a00700 1 -- 172.21.15.44:0/3511248639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c8010380 con 0x7fb8d8077b80 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.047+0000 7fb8dea00700 1 -- 172.21.15.44:0/3511248639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 msgr2=0x7fb8d8076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.047+0000 7fb8dea00700 1 --2- 172.21.15.44:0/3511248639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8076130 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80098e0 tx=0x7fb8c8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.048+0000 7fb8dea00700 1 -- 172.21.15.44:0/3511248639 shutdown_connections 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.048+0000 7fb8dea00700 1 --2- 172.21.15.44:0/3511248639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8076130 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.048+0000 7fb8dea00700 1 -- 172.21.15.44:0/3511248639 >> 172.21.15.44:0/3511248639 conn(0x7fb8d80fe0a0 msgr2=0x7fb8d81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:11.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.048+0000 7fb8dea00700 1 -- 172.21.15.44:0/3511248639 shutdown_connections 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.048+0000 7fb8dea00700 1 -- 172.21.15.44:0/3511248639 wait complete. 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8dea00700 1 Processor -- start 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8dea00700 1 -- start start 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8dea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8dea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d8113da0 con 0x7fb8d8077b80 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53764/0 (socket says 172.21.15.44:53764) 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.049+0000 7fb8d7e00700 1 -- 172.21.15.44:0/1957755503 learned_addr learned my addr 172.21.15.44:0/1957755503 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.050+0000 7fb8d7e00700 1 -- 172.21.15.44:0/1957755503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c80094f0 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.050+0000 7fb8d7e00700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8113860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8006fd0 tx=0x7fb8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.051+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c801a7d0 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.051+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d8113fa0 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.051+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb8c801adc0 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.051+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c8021360 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.051+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d8114b10 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.052+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d805f5f0 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.052+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fb8c801a930 con 0x7fb8d8077b80 2024-04-16T11:31:11.985 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.053+0000 7fb8cfe00700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fb8bc0368a0 0x7fb8bc038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.053+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb8c804a0e0 con 0x7fb8d8077b80 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.053+0000 7fb8d7400700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fb8bc0368a0 0x7fb8bc038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.054+0000 7fb8d7400700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fb8bc0368a0 0x7fb8bc038d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009990 tx=0x7fb8c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.058+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb8c807c250 con 0x7fb8d8077b80 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.275+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8d80ff240 con 0x7fb8bc0368a0 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.277+0000 7fb8cfe00700 1 -- 172.21.15.44:0/1957755503 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb8d80ff240 con 0x7fb8bc0368a0 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.279+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fb8bc0368a0 msgr2=0x7fb8bc038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.279+0000 7fb8dea00700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fb8bc0368a0 0x7fb8bc038d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009990 tx=0x7fb8c4006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.279+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 msgr2=0x7fb8d8113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.279+0000 7fb8dea00700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8113860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8006fd0 tx=0x7fb8c8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.280+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 shutdown_connections 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.280+0000 7fb8dea00700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fb8bc0368a0 0x7fb8bc038d50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.280+0000 7fb8dea00700 1 --2- 172.21.15.44:0/1957755503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb8d8077b80 0x7fb8d8113860 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.280+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 >> 172.21.15.44:0/1957755503 conn(0x7fb8d80fe0a0 msgr2=0x7fb8d80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:11.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.280+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 shutdown_connections 2024-04-16T11:31:11.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:11.280+0000 7fb8dea00700 1 -- 172.21.15.44:0/1957755503 wait complete. 2024-04-16T11:31:11.987 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-16T11:31:11.987 INFO:teuthology.orchestra.run.smithi044.stdout:Adding key to root@localhost authorized_keys... 2024-04-16T11:31:11.988 INFO:teuthology.orchestra.run.smithi044.stdout:Adding host smithi044... 2024-04-16T11:31:12.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:12 smithi044 ceph-mon[28731]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:13.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:13 smithi044 ceph-mon[28731]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:15.204 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:15 smithi044 ceph-mon[28731]: Deploying cephadm binary to smithi044 2024-04-16T11:31:17.958 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Added host 'smithi044' with addr '172.21.15.44' 2024-04-16T11:31:17.958 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.681+0000 7f954ec00700 1 Processor -- start 2024-04-16T11:31:17.958 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.681+0000 7f954ec00700 1 -- start start 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.682+0000 7f954ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f9548107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.682+0000 7f954ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9548076fb0 con 0x7f9548076ba0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.683+0000 7f9547e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f9548107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.683+0000 7f9547e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f9548107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56690/0 (socket says 172.21.15.44:56690) 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.683+0000 7f9547e00700 1 -- 172.21.15.44:0/1589916013 learned_addr learned my addr 172.21.15.44:0/1589916013 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.683+0000 7f9547e00700 1 -- 172.21.15.44:0/1589916013 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9548103360 con 0x7f9548076ba0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.683+0000 7f9547e00700 1 --2- 172.21.15.44:0/1589916013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f9548107970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95380098e0 tx=0x7f9538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc2018fd6e3f26f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.684+0000 7f9546a00700 1 -- 172.21.15.44:0/1589916013 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953800b860 con 0x7f9548076ba0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.684+0000 7f9546a00700 1 -- 172.21.15.44:0/1589916013 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f953800be50 con 0x7f9548076ba0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.684+0000 7f9546a00700 1 -- 172.21.15.44:0/1589916013 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9538010380 con 0x7f9548076ba0 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.684+0000 7f954ec00700 1 -- 172.21.15.44:0/1589916013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 msgr2=0x7f9548107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.684+0000 7f954ec00700 1 --2- 172.21.15.44:0/1589916013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f9548107970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95380098e0 tx=0x7f9538009250 comp rx=0 tx=0).stop 2024-04-16T11:31:17.959 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.685+0000 7f954ec00700 1 -- 172.21.15.44:0/1589916013 shutdown_connections 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.685+0000 7f954ec00700 1 --2- 172.21.15.44:0/1589916013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f9548107970 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.685+0000 7f954ec00700 1 -- 172.21.15.44:0/1589916013 >> 172.21.15.44:0/1589916013 conn(0x7f95480fcfc0 msgr2=0x7f95480ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.685+0000 7f954ec00700 1 -- 172.21.15.44:0/1589916013 shutdown_connections 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.685+0000 7f954ec00700 1 -- 172.21.15.44:0/1589916013 wait complete. 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.686+0000 7f954ec00700 1 Processor -- start 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.686+0000 7f954ec00700 1 -- start start 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.686+0000 7f954ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f95481137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.686+0000 7f954ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9548113cf0 con 0x7f9548076ba0 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.687+0000 7f9547e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f95481137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.687+0000 7f9547e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f95481137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56700/0 (socket says 172.21.15.44:56700) 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.687+0000 7f9547e00700 1 -- 172.21.15.44:0/3623735048 learned_addr learned my addr 172.21.15.44:0/3623735048 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.687+0000 7f9547e00700 1 -- 172.21.15.44:0/3623735048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95380094f0 con 0x7f9548076ba0 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.687+0000 7f9547e00700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f95481137b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f953800ba90 tx=0x7f9538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.687+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953801a8a0 con 0x7f9548076ba0 2024-04-16T11:31:17.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.688+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f953801ae90 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.688+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9538019e80 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.688+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9548113f50 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.688+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9548114a60 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.689+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f9538019730 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.689+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954805f5f0 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.689+0000 7f953fe00700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f952c0368a0 0x7f952c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.689+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f953804a170 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.690+0000 7f9547400700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f952c0368a0 0x7f952c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.690+0000 7f9547400700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f952c0368a0 0x7f952c038d50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9534009990 tx=0x7f9534006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.694+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9538021d30 con 0x7f9548076ba0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:12.913+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}) v1 -- 0x7f95480fe110 con 0x7f952c0368a0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.273+0000 7f953fe00700 1 -- 172.21.15.44:0/3623735048 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f95480fe110 con 0x7f952c0368a0 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.277+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f952c0368a0 msgr2=0x7f952c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.277+0000 7f954ec00700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f952c0368a0 0x7f952c038d50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9534009990 tx=0x7f9534006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:17.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.277+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 msgr2=0x7f95481137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.277+0000 7f954ec00700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f95481137b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f953800ba90 tx=0x7f9538009250 comp rx=0 tx=0).stop 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.278+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 shutdown_connections 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.278+0000 7f954ec00700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f952c0368a0 0x7f952c038d50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.278+0000 7f954ec00700 1 --2- 172.21.15.44:0/3623735048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9548076ba0 0x7f95481137b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.278+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 >> 172.21.15.44:0/3623735048 conn(0x7f95480fcfc0 msgr2=0x7f95480fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.278+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 shutdown_connections 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:17.278+0000 7f954ec00700 1 -- 172.21.15.44:0/3623735048 wait complete. 2024-04-16T11:31:17.962 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying mon service with default placement... 2024-04-16T11:31:18.609 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:18 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:18.609 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:18 smithi044 ceph-mon[28731]: Added host smithi044 2024-04-16T11:31:18.609 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:18 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:20.041 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-16T11:31:20.041 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.616+0000 7f7feb200700 1 Processor -- start 2024-04-16T11:31:20.042 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.616+0000 7f7feb200700 1 -- start start 2024-04-16T11:31:20.042 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.617+0000 7f7feb200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec104e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:20.042 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.617+0000 7f7feb200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec105420 con 0x7f7fec104a40 2024-04-16T11:31:20.042 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.617+0000 7f7fe9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec104e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:20.042 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.617+0000 7f7fe9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec104e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56704/0 (socket says 172.21.15.44:56704) 2024-04-16T11:31:20.043 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.617+0000 7f7fe9e00700 1 -- 172.21.15.44:0/1337396266 learned_addr learned my addr 172.21.15.44:0/1337396266 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:20.043 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.618+0000 7f7fe9e00700 1 -- 172.21.15.44:0/1337396266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec105560 con 0x7f7fec104a40 2024-04-16T11:31:20.043 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.618+0000 7f7fe9e00700 1 --2- 172.21.15.44:0/1337396266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec104e50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0098e0 tx=0x7f7fdc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538841aedf047d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:20.044 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7fe8a00700 1 -- 172.21.15.44:0/1337396266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fdc00b860 con 0x7f7fec104a40 2024-04-16T11:31:20.044 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7fe8a00700 1 -- 172.21.15.44:0/1337396266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f7fdc00be50 con 0x7f7fec104a40 2024-04-16T11:31:20.044 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7fe8a00700 1 -- 172.21.15.44:0/1337396266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fdc010380 con 0x7f7fec104a40 2024-04-16T11:31:20.044 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7feb200700 1 -- 172.21.15.44:0/1337396266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 msgr2=0x7f7fec104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:20.044 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7feb200700 1 --2- 172.21.15.44:0/1337396266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec104e50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0098e0 tx=0x7f7fdc009250 comp rx=0 tx=0).stop 2024-04-16T11:31:20.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7feb200700 1 -- 172.21.15.44:0/1337396266 shutdown_connections 2024-04-16T11:31:20.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7feb200700 1 --2- 172.21.15.44:0/1337396266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec104e50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:20.045 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7feb200700 1 -- 172.21.15.44:0/1337396266 >> 172.21.15.44:0/1337396266 conn(0x7f7fec078160 msgr2=0x7f7fec078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:20.046 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.619+0000 7f7feb200700 1 -- 172.21.15.44:0/1337396266 shutdown_connections 2024-04-16T11:31:20.046 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.620+0000 7f7feb200700 1 -- 172.21.15.44:0/1337396266 wait complete. 2024-04-16T11:31:20.046 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.620+0000 7f7feb200700 1 Processor -- start 2024-04-16T11:31:20.046 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.620+0000 7f7feb200700 1 -- start start 2024-04-16T11:31:20.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.621+0000 7f7feb200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:20.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.621+0000 7f7feb200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec113ac0 con 0x7f7fec104a40 2024-04-16T11:31:20.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.621+0000 7f7fe9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:20.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.621+0000 7f7fe9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec113580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56720/0 (socket says 172.21.15.44:56720) 2024-04-16T11:31:20.047 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.621+0000 7f7fe9e00700 1 -- 172.21.15.44:0/4213693025 learned_addr learned my addr 172.21.15.44:0/4213693025 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:20.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.621+0000 7f7fe9e00700 1 -- 172.21.15.44:0/4213693025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0094f0 con 0x7f7fec104a40 2024-04-16T11:31:20.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.622+0000 7f7fe9e00700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec113580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00ba90 tx=0x7f7fdc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:20.048 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.622+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fdc01a8a0 con 0x7f7fec104a40 2024-04-16T11:31:20.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.622+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f7fdc01ae90 con 0x7f7fec104a40 2024-04-16T11:31:20.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.622+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec113cc0 con 0x7f7fec104a40 2024-04-16T11:31:20.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.622+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fdc019e80 con 0x7f7fec104a40 2024-04-16T11:31:20.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.622+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec114830 con 0x7f7fec104a40 2024-04-16T11:31:20.049 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.623+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f7fdc018460 con 0x7f7fec104a40 2024-04-16T11:31:20.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.623+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc0052e0 con 0x7f7fec104a40 2024-04-16T11:31:20.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.624+0000 7f7fe2000700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7fd00368a0 0x7f7fd0038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:20.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.624+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7fdc04a0c0 con 0x7f7fec104a40 2024-04-16T11:31:20.050 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.624+0000 7f7fe9400700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7fd00368a0 0x7f7fd0038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:20.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.625+0000 7f7fe9400700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7fd00368a0 0x7f7fd0038d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009990 tx=0x7f7fd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:20.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.629+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7fdc010d50 con 0x7f7fec104a40 2024-04-16T11:31:20.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.868+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fcc000bf0 con 0x7f7fd00368a0 2024-04-16T11:31:20.051 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.875+0000 7f7fe2000700 1 -- 172.21.15.44:0/4213693025 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7fcc000bf0 con 0x7f7fd00368a0 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.878+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7fd00368a0 msgr2=0x7f7fd0038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.878+0000 7f7feb200700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7fd00368a0 0x7f7fd0038d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009990 tx=0x7f7fd8006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.878+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 msgr2=0x7f7fec113580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.878+0000 7f7feb200700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec113580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00ba90 tx=0x7f7fdc009250 comp rx=0 tx=0).stop 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.879+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 shutdown_connections 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.879+0000 7f7feb200700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f7fd00368a0 0x7f7fd0038d50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.879+0000 7f7feb200700 1 --2- 172.21.15.44:0/4213693025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7fec104a40 0x7f7fec113580 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:20.052 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.879+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 >> 172.21.15.44:0/4213693025 conn(0x7f7fec078160 msgr2=0x7f7fec102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:20.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.879+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 shutdown_connections 2024-04-16T11:31:20.053 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:18.879+0000 7f7feb200700 1 -- 172.21.15.44:0/4213693025 wait complete. 2024-04-16T11:31:20.053 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying mgr service with default placement... 2024-04-16T11:31:20.143 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:19 smithi044 ceph-mon[28731]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:20.143 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:19 smithi044 ceph-mon[28731]: Saving service mon spec with placement count:5 2024-04-16T11:31:20.143 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:19 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:21.710 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:21 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:21.710 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:21 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:21.710 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:21 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:22.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-16T11:31:22.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.117+0000 7fdb0ea00700 1 Processor -- start 2024-04-16T11:31:22.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.118+0000 7fdb0ea00700 1 -- start start 2024-04-16T11:31:22.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.118+0000 7fdb0ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb08105040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:22.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.118+0000 7fdb0ea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb08105610 con 0x7fdb08104c30 2024-04-16T11:31:22.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.119+0000 7fdb07e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb08105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:22.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.119+0000 7fdb07e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb08105040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56730/0 (socket says 172.21.15.44:56730) 2024-04-16T11:31:22.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.119+0000 7fdb07e00700 1 -- 172.21.15.44:0/2697782422 learned_addr learned my addr 172.21.15.44:0/2697782422 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:22.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.119+0000 7fdb07e00700 1 -- 172.21.15.44:0/2697782422 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08105750 con 0x7fdb08104c30 2024-04-16T11:31:22.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.120+0000 7fdb07e00700 1 --2- 172.21.15.44:0/2697782422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb08105040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80098e0 tx=0x7fdaf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2e14426db2a294 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:22.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.120+0000 7fdb06a00700 1 -- 172.21.15.44:0/2697782422 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaf800b860 con 0x7fdb08104c30 2024-04-16T11:31:22.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.120+0000 7fdb06a00700 1 -- 172.21.15.44:0/2697782422 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaf800be90 con 0x7fdb08104c30 2024-04-16T11:31:22.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.120+0000 7fdb06a00700 1 -- 172.21.15.44:0/2697782422 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaf80103c0 con 0x7fdb08104c30 2024-04-16T11:31:22.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2697782422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 msgr2=0x7fdb08105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:22.086 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 --2- 172.21.15.44:0/2697782422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb08105040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80098e0 tx=0x7fdaf8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2697782422 shutdown_connections 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 --2- 172.21.15.44:0/2697782422 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb08105040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2697782422 >> 172.21.15.44:0/2697782422 conn(0x7fdb08100590 msgr2=0x7fdb08102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2697782422 shutdown_connections 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.121+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2697782422 wait complete. 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.122+0000 7fdb0ea00700 1 Processor -- start 2024-04-16T11:31:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.122+0000 7fdb0ea00700 1 -- start start 2024-04-16T11:31:22.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.122+0000 7fdb0ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb081137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:22.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.122+0000 7fdb0ea00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb08113cf0 con 0x7fdb08104c30 2024-04-16T11:31:22.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.123+0000 7fdb07e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb081137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:22.088 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.123+0000 7fdb07e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb081137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56736/0 (socket says 172.21.15.44:56736) 2024-04-16T11:31:22.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.123+0000 7fdb07e00700 1 -- 172.21.15.44:0/2784899585 learned_addr learned my addr 172.21.15.44:0/2784899585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:22.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.123+0000 7fdb07e00700 1 -- 172.21.15.44:0/2784899585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf80094f0 con 0x7fdb08104c30 2024-04-16T11:31:22.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.123+0000 7fdb07e00700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb081137b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8006b00 tx=0x7fdaf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:22.089 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.124+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaf8010840 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.124+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaf8010e70 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.124+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaf801ae70 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.124+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb08113ef0 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.124+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb08114a60 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.125+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0805f5f0 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.125+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fdaf801a780 con 0x7fdb08104c30 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.126+0000 7fdaffe00700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fdaec036850 0x7fdaec038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:22.090 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.126+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdaf804ac10 con 0x7fdb08104c30 2024-04-16T11:31:22.092 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.126+0000 7fdb07400700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fdaec036850 0x7fdaec038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:22.092 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.127+0000 7fdb07400700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fdaec036850 0x7fdaec038d00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009990 tx=0x7fdaf4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:22.092 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.133+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdaf801a430 con 0x7fdb08104c30 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.383+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb08102690 con 0x7fdaec036850 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.391+0000 7fdaffe00700 1 -- 172.21.15.44:0/2784899585 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdb08102690 con 0x7fdaec036850 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fdaec036850 msgr2=0x7fdaec038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fdaec036850 0x7fdaec038d00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009990 tx=0x7fdaf4006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 msgr2=0x7fdb081137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb081137b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8006b00 tx=0x7fdaf8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 shutdown_connections 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fdaec036850 0x7fdaec038d00 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009990 tx=0x7fdaf4006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 --2- 172.21.15.44:0/2784899585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb08104c30 0x7fdb081137b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.393+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 >> 172.21.15.44:0/2784899585 conn(0x7fdb08100590 msgr2=0x7fdb08101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.394+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 shutdown_connections 2024-04-16T11:31:22.093 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:21.394+0000 7fdb0ea00700 1 -- 172.21.15.44:0/2784899585 wait complete. 2024-04-16T11:31:22.094 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying crash service with default placement... 2024-04-16T11:31:22.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:22 smithi044 ceph-mon[28731]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:22.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:22 smithi044 ceph-mon[28731]: Saving service mgr spec with placement count:2 2024-04-16T11:31:24.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-16T11:31:24.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.722+0000 7f328c800700 1 Processor -- start 2024-04-16T11:31:24.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.723+0000 7f328c800700 1 -- start start 2024-04-16T11:31:24.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.723+0000 7f328c800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 0x7f3284102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:24.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.723+0000 7f328c800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284103250 con 0x7f3284102900 2024-04-16T11:31:24.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.724+0000 7f3289e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 0x7f3284102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:24.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.724+0000 7f3289e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 0x7f3284102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56626/0 (socket says 172.21.15.44:56626) 2024-04-16T11:31:24.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.724+0000 7f3289e00700 1 -- 172.21.15.44:0/2674210786 learned_addr learned my addr 172.21.15.44:0/2674210786 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.724+0000 7f3289e00700 1 -- 172.21.15.44:0/2674210786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284103390 con 0x7f3284102900 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.725+0000 7f3289e00700 1 --2- 172.21.15.44:0/2674210786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 0x7f3284102d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32780098e0 tx=0x7f3278009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8746a6a0c1b2a232 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.725+0000 7f3288a00700 1 -- 172.21.15.44:0/2674210786 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327800b860 con 0x7f3284102900 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.725+0000 7f3288a00700 1 -- 172.21.15.44:0/2674210786 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f327800be90 con 0x7f3284102900 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.725+0000 7f3288a00700 1 -- 172.21.15.44:0/2674210786 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32780103c0 con 0x7f3284102900 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.726+0000 7f328c800700 1 -- 172.21.15.44:0/2674210786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 msgr2=0x7f3284102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.726+0000 7f328c800700 1 --2- 172.21.15.44:0/2674210786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 0x7f3284102d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32780098e0 tx=0x7f3278009250 comp rx=0 tx=0).stop 2024-04-16T11:31:24.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.726+0000 7f328c800700 1 -- 172.21.15.44:0/2674210786 shutdown_connections 2024-04-16T11:31:24.123 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.726+0000 7f328c800700 1 --2- 172.21.15.44:0/2674210786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3284102900 0x7f3284102d10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.726+0000 7f328c800700 1 -- 172.21.15.44:0/2674210786 >> 172.21.15.44:0/2674210786 conn(0x7f32840fe0a0 msgr2=0x7f32841004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.726+0000 7f328c800700 1 -- 172.21.15.44:0/2674210786 shutdown_connections 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.727+0000 7f328c800700 1 -- 172.21.15.44:0/2674210786 wait complete. 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f328c800700 1 Processor -- start 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f328c800700 1 -- start start 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f328c800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 0x7f3284113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f328c800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284114310 con 0x7f32841139c0 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f3289e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 0x7f3284113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f3289e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 0x7f3284113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56630/0 (socket says 172.21.15.44:56630) 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.728+0000 7f3289e00700 1 -- 172.21.15.44:0/2135647881 learned_addr learned my addr 172.21.15.44:0/2135647881 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.729+0000 7f3289e00700 1 -- 172.21.15.44:0/2135647881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780094f0 con 0x7f32841139c0 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.729+0000 7f3289e00700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 0x7f3284113dd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3278011c10 tx=0x7f3278009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.730+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3278011890 con 0x7f32841139c0 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.730+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f327801f070 con 0x7f32841139c0 2024-04-16T11:31:24.124 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.730+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3284114510 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.730+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3278026990 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.730+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3284114ef0 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.731+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f3278026b40 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.731+0000 7f327e000700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f326c0368a0 0x7f326c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.731+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3278021070 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.731+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328404bed0 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.732+0000 7f3289400700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f326c0368a0 0x7f326c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.732+0000 7f3289400700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f326c0368a0 0x7f326c038d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3274009990 tx=0x7f3274006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.740+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f327801b440 con 0x7f32841139c0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.977+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f32840ff260 con 0x7f326c0368a0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.984+0000 7f327e000700 1 -- 172.21.15.44:0/2135647881 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f32840ff260 con 0x7f326c0368a0 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f326c0368a0 msgr2=0x7f326c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:24.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f326c0368a0 0x7f326c038d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3274009990 tx=0x7f3274006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:24.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 msgr2=0x7f3284113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 0x7f3284113dd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3278011c10 tx=0x7f3278009250 comp rx=0 tx=0).stop 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 shutdown_connections 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f326c0368a0 0x7f326c038d50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 --2- 172.21.15.44:0/2135647881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32841139c0 0x7f3284113dd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 >> 172.21.15.44:0/2135647881 conn(0x7f32840fe0a0 msgr2=0x7f32840feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 shutdown_connections 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:22.987+0000 7f328c800700 1 -- 172.21.15.44:0/2135647881 wait complete. 2024-04-16T11:31:24.127 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying prometheus service with default placement... 2024-04-16T11:31:24.295 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:23 smithi044 ceph-mon[28731]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:24.295 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:23 smithi044 ceph-mon[28731]: Saving service crash spec with placement * 2024-04-16T11:31:24.295 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:23 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:25.417 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:24 smithi044 ceph-mon[28731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:25.418 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:24 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.777+0000 7f085f800700 1 Processor -- start 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.777+0000 7f085f800700 1 -- start start 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.777+0000 7f085f800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.777+0000 7f085f800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0858077fb0 con 0x7f0858077ba0 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.778+0000 7f085ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.778+0000 7f085ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56636/0 (socket says 172.21.15.44:56636) 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.778+0000 7f085ce00700 1 -- 172.21.15.44:0/1077754878 learned_addr learned my addr 172.21.15.44:0/1077754878 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.779+0000 7f085ce00700 1 -- 172.21.15.44:0/1077754878 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0858076690 con 0x7f0858077ba0 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.779+0000 7f085ce00700 1 --2- 172.21.15.44:0/1077754878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858076150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f084c0098e0 tx=0x7f084c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6395b67666bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.779+0000 7f0857400700 1 -- 172.21.15.44:0/1077754878 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f084c00b860 con 0x7f0858077ba0 2024-04-16T11:31:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.779+0000 7f0857400700 1 -- 172.21.15.44:0/1077754878 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f084c00be90 con 0x7f0858077ba0 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.779+0000 7f0857400700 1 -- 172.21.15.44:0/1077754878 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f084c0103c0 con 0x7f0858077ba0 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.780+0000 7f085f800700 1 -- 172.21.15.44:0/1077754878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 msgr2=0x7f0858076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.780+0000 7f085f800700 1 --2- 172.21.15.44:0/1077754878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858076150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f084c0098e0 tx=0x7f084c009250 comp rx=0 tx=0).stop 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.780+0000 7f085f800700 1 -- 172.21.15.44:0/1077754878 shutdown_connections 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.780+0000 7f085f800700 1 --2- 172.21.15.44:0/1077754878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858076150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.780+0000 7f085f800700 1 -- 172.21.15.44:0/1077754878 >> 172.21.15.44:0/1077754878 conn(0x7f08580fe0a0 msgr2=0x7f08581004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.781+0000 7f085f800700 1 -- 172.21.15.44:0/1077754878 shutdown_connections 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.781+0000 7f085f800700 1 -- 172.21.15.44:0/1077754878 wait complete. 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.782+0000 7f085f800700 1 Processor -- start 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.782+0000 7f085f800700 1 -- start start 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.782+0000 7f085f800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.782+0000 7f085f800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0858113dc0 con 0x7f0858077ba0 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.783+0000 7f085ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.783+0000 7f085ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858113880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56652/0 (socket says 172.21.15.44:56652) 2024-04-16T11:31:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.783+0000 7f085ce00700 1 -- 172.21.15.44:0/1638414180 learned_addr learned my addr 172.21.15.44:0/1638414180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.783+0000 7f085ce00700 1 -- 172.21.15.44:0/1638414180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0094f0 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.784+0000 7f085ce00700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858113880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f084c0105b0 tx=0x7f084c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.784+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f084c010840 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.784+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f084c010e70 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.784+0000 7f085f800700 1 -- 172.21.15.44:0/1638414180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0858113fc0 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.784+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f084c01ae30 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.784+0000 7f085f800700 1 -- 172.21.15.44:0/1638414180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0858114b30 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.786+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f084c02e440 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.786+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085805f5f0 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.786+0000 7f0854c00700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f08400368a0 0x7f0840038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.786+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f084c04a5f0 con 0x7f0858077ba0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.787+0000 7f0857e00700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f08400368a0 0x7f0840038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.788+0000 7f0857e00700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f08400368a0 0x7f0840038d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0844009990 tx=0x7f0844006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:25.721 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:24.792+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f084c02c930 con 0x7f0858077ba0 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.026+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f0858108b30 con 0x7f08400368a0 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.036+0000 7f0854c00700 1 -- 172.21.15.44:0/1638414180 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f0858108b30 con 0x7f08400368a0 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.038+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f08400368a0 msgr2=0x7f0840038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.038+0000 7f084a000700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f08400368a0 0x7f0840038d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0844009990 tx=0x7f0844006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.038+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 msgr2=0x7f0858113880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.038+0000 7f084a000700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858113880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f084c0105b0 tx=0x7f084c009250 comp rx=0 tx=0).stop 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.039+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 shutdown_connections 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.039+0000 7f084a000700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f08400368a0 0x7f0840038d50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.039+0000 7f084a000700 1 --2- 172.21.15.44:0/1638414180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0858077ba0 0x7f0858113880 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.039+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 >> 172.21.15.44:0/1638414180 conn(0x7f08580fe0a0 msgr2=0x7f08580feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.039+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 shutdown_connections 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:25.039+0000 7f084a000700 1 -- 172.21.15.44:0/1638414180 wait complete. 2024-04-16T11:31:25.722 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying grafana service with default placement... 2024-04-16T11:31:26.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:26 smithi044 ceph-mon[28731]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:26.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:26 smithi044 ceph-mon[28731]: Saving service prometheus spec with placement count:1 2024-04-16T11:31:26.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:26 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:26.310 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:26 smithi044 ceph-mon[28731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:26.310 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:26 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:27.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:27 smithi044 ceph-mon[28731]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:27.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:27 smithi044 ceph-mon[28731]: Saving service grafana spec with placement count:1 2024-04-16T11:31:27.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:27 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:28.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-16T11:31:28.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.373+0000 7f5b2e800700 1 Processor -- start 2024-04-16T11:31:28.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.374+0000 7f5b2e800700 1 -- start start 2024-04-16T11:31:28.289 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.374+0000 7f5b2e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:28.291 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.374+0000 7f5b2e800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28077fb0 con 0x7f5b28077ba0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.375+0000 7f5b27400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.375+0000 7f5b27400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56666/0 (socket says 172.21.15.44:56666) 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.375+0000 7f5b27400700 1 -- 172.21.15.44:0/3222798924 learned_addr learned my addr 172.21.15.44:0/3222798924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.375+0000 7f5b27400700 1 -- 172.21.15.44:0/3222798924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28076690 con 0x7f5b28077ba0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.376+0000 7f5b27400700 1 --2- 172.21.15.44:0/3222798924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28076150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5b180098e0 tx=0x7f5b18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcf75829b1cd2cf server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.376+0000 7f5b26000700 1 -- 172.21.15.44:0/3222798924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1800b860 con 0x7f5b28077ba0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.376+0000 7f5b26000700 1 -- 172.21.15.44:0/3222798924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b1800be90 con 0x7f5b28077ba0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.376+0000 7f5b26000700 1 -- 172.21.15.44:0/3222798924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b180103c0 con 0x7f5b28077ba0 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 -- 172.21.15.44:0/3222798924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 msgr2=0x7f5b28076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 --2- 172.21.15.44:0/3222798924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28076150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5b180098e0 tx=0x7f5b18009250 comp rx=0 tx=0).stop 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 -- 172.21.15.44:0/3222798924 shutdown_connections 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 --2- 172.21.15.44:0/3222798924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28076150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:28.292 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 -- 172.21.15.44:0/3222798924 >> 172.21.15.44:0/3222798924 conn(0x7f5b280fe0a0 msgr2=0x7f5b281004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:28.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 -- 172.21.15.44:0/3222798924 shutdown_connections 2024-04-16T11:31:28.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.377+0000 7f5b2e800700 1 -- 172.21.15.44:0/3222798924 wait complete. 2024-04-16T11:31:28.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.378+0000 7f5b2e800700 1 Processor -- start 2024-04-16T11:31:28.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.378+0000 7f5b2e800700 1 -- start start 2024-04-16T11:31:28.293 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.379+0000 7f5b2e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:28.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.379+0000 7f5b2e800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28113dc0 con 0x7f5b28077ba0 2024-04-16T11:31:28.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.379+0000 7f5b27400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:28.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.379+0000 7f5b27400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28113880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56678/0 (socket says 172.21.15.44:56678) 2024-04-16T11:31:28.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.379+0000 7f5b27400700 1 -- 172.21.15.44:0/2205826185 learned_addr learned my addr 172.21.15.44:0/2205826185 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.379+0000 7f5b27400700 1 -- 172.21.15.44:0/2205826185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b180094f0 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.380+0000 7f5b27400700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28113880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5b180094c0 tx=0x7f5b18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.380+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b180118f0 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.380+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b18010d60 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.380+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b28113fc0 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.380+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1801a430 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.380+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b28114b30 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.381+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2805f5f0 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.382+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f5b18024440 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.382+0000 7f5b1f400700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f5b0c0368a0 0x7f5b0c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.382+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5b1804a510 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.383+0000 7f5b26a00700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f5b0c0368a0 0x7f5b0c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.383+0000 7f5b26a00700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f5b0c0368a0 0x7f5b0c038d50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009990 tx=0x7f5b14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.388+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5b18024760 con 0x7f5b28077ba0 2024-04-16T11:31:28.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.632+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b280ff230 con 0x7f5b0c0368a0 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.640+0000 7f5b1f400700 1 -- 172.21.15.44:0/2205826185 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5b280ff230 con 0x7f5b0c0368a0 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.642+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f5b0c0368a0 msgr2=0x7f5b0c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.642+0000 7f5b2e800700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f5b0c0368a0 0x7f5b0c038d50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009990 tx=0x7f5b14006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 msgr2=0x7f5b28113880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28113880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5b180094c0 tx=0x7f5b18009250 comp rx=0 tx=0).stop 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 shutdown_connections 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f5b0c0368a0 0x7f5b0c038d50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 --2- 172.21.15.44:0/2205826185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b28077ba0 0x7f5b28113880 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 >> 172.21.15.44:0/2205826185 conn(0x7f5b280fe0a0 msgr2=0x7f5b280feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 shutdown_connections 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:26.643+0000 7f5b2e800700 1 -- 172.21.15.44:0/2205826185 wait complete. 2024-04-16T11:31:28.298 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying node-exporter service with default placement... 2024-04-16T11:31:28.889 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:28 smithi044 ceph-mon[28731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:29.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-16T11:31:29.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.959+0000 7f040cc00700 1 Processor -- start 2024-04-16T11:31:29.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.959+0000 7f040cc00700 1 -- start start 2024-04-16T11:31:29.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.960+0000 7f040cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f0408105000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:29.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.960+0000 7f040cc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081055d0 con 0x7f0408104bf0 2024-04-16T11:31:29.904 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.960+0000 7f0405e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f0408105000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.960+0000 7f0405e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f0408105000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56684/0 (socket says 172.21.15.44:56684) 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.960+0000 7f0405e00700 1 -- 172.21.15.44:0/2439005356 learned_addr learned my addr 172.21.15.44:0/2439005356 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.961+0000 7f0405e00700 1 -- 172.21.15.44:0/2439005356 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408105710 con 0x7f0408104bf0 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.961+0000 7f0405e00700 1 --2- 172.21.15.44:0/2439005356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f0408105000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f03f80098e0 tx=0x7f03f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5507ed9da31a450 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.961+0000 7f0404a00700 1 -- 172.21.15.44:0/2439005356 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f800b860 con 0x7f0408104bf0 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f0404a00700 1 -- 172.21.15.44:0/2439005356 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03f800be90 con 0x7f0408104bf0 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f0404a00700 1 -- 172.21.15.44:0/2439005356 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f80103c0 con 0x7f0408104bf0 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f040cc00700 1 -- 172.21.15.44:0/2439005356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 msgr2=0x7f0408105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f040cc00700 1 --2- 172.21.15.44:0/2439005356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f0408105000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f03f80098e0 tx=0x7f03f8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f040cc00700 1 -- 172.21.15.44:0/2439005356 shutdown_connections 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f040cc00700 1 --2- 172.21.15.44:0/2439005356 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f0408105000 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.962+0000 7f040cc00700 1 -- 172.21.15.44:0/2439005356 >> 172.21.15.44:0/2439005356 conn(0x7f0408078320 msgr2=0x7f0408078720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.963+0000 7f040cc00700 1 -- 172.21.15.44:0/2439005356 shutdown_connections 2024-04-16T11:31:29.905 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.963+0000 7f040cc00700 1 -- 172.21.15.44:0/2439005356 wait complete. 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.963+0000 7f040cc00700 1 Processor -- start 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.964+0000 7f040cc00700 1 -- start start 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.964+0000 7f040cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f04081137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.964+0000 7f040cc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408113cf0 con 0x7f0408104bf0 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.964+0000 7f0405e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f04081137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.964+0000 7f0405e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f04081137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56702/0 (socket says 172.21.15.44:56702) 2024-04-16T11:31:29.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.964+0000 7f0405e00700 1 -- 172.21.15.44:0/476595213 learned_addr learned my addr 172.21.15.44:0/476595213 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f0405e00700 1 -- 172.21.15.44:0/476595213 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80094f0 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f0405e00700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f04081137b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03f80090b0 tx=0x7f03f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f8010840 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03f8010e70 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0408113ef0 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f8024e40 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.965+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0408114a60 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.966+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f03f8024430 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.967+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040805f5f0 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.967+0000 7f03fe000700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f03ec036850 0x7f03ec038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.967+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f03f804a860 con 0x7f0408104bf0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.967+0000 7f0405400700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f03ec036850 0x7f03ec038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.968+0000 7f0405400700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f03ec036850 0x7f03ec038d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009990 tx=0x7f03f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:29.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:28.972+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f03f802c910 con 0x7f0408104bf0 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.201+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f0408102b70 con 0x7f03ec036850 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.212+0000 7f03fe000700 1 -- 172.21.15.44:0/476595213 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f0408102b70 con 0x7f03ec036850 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f03ec036850 msgr2=0x7f03ec038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f03ec036850 0x7f03ec038d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009990 tx=0x7f03f4006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 msgr2=0x7f04081137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f04081137b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03f80090b0 tx=0x7f03f8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 shutdown_connections 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f03ec036850 0x7f03ec038d00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 --2- 172.21.15.44:0/476595213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0408104bf0 0x7f04081137b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.215+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 >> 172.21.15.44:0/476595213 conn(0x7f0408078320 msgr2=0x7f0408102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.216+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 shutdown_connections 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:29.216+0000 7f040cc00700 1 -- 172.21.15.44:0/476595213 wait complete. 2024-04-16T11:31:29.911 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying alertmanager service with default placement... 2024-04-16T11:31:30.471 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:30 smithi044 ceph-mon[28731]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:30.471 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:30 smithi044 ceph-mon[28731]: Saving service node-exporter spec with placement * 2024-04-16T11:31:30.471 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:30 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:30.471 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:30 smithi044 ceph-mon[28731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:31.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-16T11:31:31.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.553+0000 7fa32a000700 1 Processor -- start 2024-04-16T11:31:31.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.554+0000 7fa32a000700 1 -- start start 2024-04-16T11:31:31.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.554+0000 7fa32a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa324105000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:31.482 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.554+0000 7fa32a000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3241055d0 con 0x7fa324104bf0 2024-04-16T11:31:31.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.554+0000 7fa323400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa324105000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:31.483 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.554+0000 7fa323400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa324105000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56710/0 (socket says 172.21.15.44:56710) 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.554+0000 7fa323400700 1 -- 172.21.15.44:0/4107662061 learned_addr learned my addr 172.21.15.44:0/4107662061 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.555+0000 7fa323400700 1 -- 172.21.15.44:0/4107662061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324105710 con 0x7fa324104bf0 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.555+0000 7fa323400700 1 --2- 172.21.15.44:0/4107662061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa324105000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3140098e0 tx=0x7fa314009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad46d865573c924 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.556+0000 7fa322000700 1 -- 172.21.15.44:0/4107662061 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa31400b860 con 0x7fa324104bf0 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.556+0000 7fa322000700 1 -- 172.21.15.44:0/4107662061 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa31400be90 con 0x7fa324104bf0 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.556+0000 7fa322000700 1 -- 172.21.15.44:0/4107662061 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3140103c0 con 0x7fa324104bf0 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.556+0000 7fa32a000700 1 -- 172.21.15.44:0/4107662061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 msgr2=0x7fa324105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.556+0000 7fa32a000700 1 --2- 172.21.15.44:0/4107662061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa324105000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3140098e0 tx=0x7fa314009250 comp rx=0 tx=0).stop 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.557+0000 7fa32a000700 1 -- 172.21.15.44:0/4107662061 shutdown_connections 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.557+0000 7fa32a000700 1 --2- 172.21.15.44:0/4107662061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa324105000 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.557+0000 7fa32a000700 1 -- 172.21.15.44:0/4107662061 >> 172.21.15.44:0/4107662061 conn(0x7fa324078320 msgr2=0x7fa324078720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.557+0000 7fa32a000700 1 -- 172.21.15.44:0/4107662061 shutdown_connections 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.557+0000 7fa32a000700 1 -- 172.21.15.44:0/4107662061 wait complete. 2024-04-16T11:31:31.484 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.558+0000 7fa32a000700 1 Processor -- start 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.558+0000 7fa32a000700 1 -- start start 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.558+0000 7fa32a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa3241137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.558+0000 7fa32a000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa324113cf0 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.558+0000 7fa323400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa3241137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.559+0000 7fa323400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa3241137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56712/0 (socket says 172.21.15.44:56712) 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.559+0000 7fa323400700 1 -- 172.21.15.44:0/1580847323 learned_addr learned my addr 172.21.15.44:0/1580847323 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.559+0000 7fa323400700 1 -- 172.21.15.44:0/1580847323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140094f0 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.559+0000 7fa323400700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa3241137b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa3140090b0 tx=0x7fa314009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.560+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa314010840 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.560+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa314010e70 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.560+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa324113ef0 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.560+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa314024e40 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.560+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa324114a60 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.561+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fa31401a4c0 con 0x7fa324104bf0 2024-04-16T11:31:31.485 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.561+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3040052e0 con 0x7fa324104bf0 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.561+0000 7fa31b400700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa308036850 0x7fa308038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.561+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa31404a840 con 0x7fa324104bf0 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.562+0000 7fa322a00700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa308036850 0x7fa308038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.563+0000 7fa322a00700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa308036850 0x7fa308038d00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa310009990 tx=0x7fa310006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.567+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa314018940 con 0x7fa324104bf0 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.795+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fa304000bf0 con 0x7fa308036850 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.805+0000 7fa31b400700 1 -- 172.21.15.44:0/1580847323 <== mgr.14120 v2:172.21.15.44:6800/2358899774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fa304000bf0 con 0x7fa308036850 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa308036850 msgr2=0x7fa308038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa308036850 0x7fa308038d00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa310009990 tx=0x7fa310006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 msgr2=0x7fa3241137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa3241137b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa3140090b0 tx=0x7fa314009250 comp rx=0 tx=0).stop 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 shutdown_connections 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa308036850 0x7fa308038d00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:31.486 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 --2- 172.21.15.44:0/1580847323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa324104bf0 0x7fa3241137b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:31.487 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.807+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 >> 172.21.15.44:0/1580847323 conn(0x7fa324078320 msgr2=0x7fa324102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:31.487 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.808+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 shutdown_connections 2024-04-16T11:31:31.487 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:30.808+0000 7fa32a000700 1 -- 172.21.15.44:0/1580847323 wait complete. 2024-04-16T11:31:32.094 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:31 smithi044 ceph-mon[28731]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:32.094 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:31 smithi044 ceph-mon[28731]: Saving service alertmanager spec with placement count:1 2024-04-16T11:31:32.094 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:31 smithi044 ceph-mon[28731]: from='mgr.14120 172.21.15.44:0/3534622485' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:33.122 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:32 smithi044 ceph-mon[28731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:33.123 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:32 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2983636317' entity='client.admin' 2024-04-16T11:31:33.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.152+0000 7fa740400700 1 Processor -- start 2024-04-16T11:31:33.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.152+0000 7fa740400700 1 -- start start 2024-04-16T11:31:33.138 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.153+0000 7fa740400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738105000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:33.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.153+0000 7fa740400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381055d0 con 0x7fa738104bf0 2024-04-16T11:31:33.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.153+0000 7fa73da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738105000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:33.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.153+0000 7fa73da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738105000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36162/0 (socket says 172.21.15.44:36162) 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.153+0000 7fa73da00700 1 -- 172.21.15.44:0/2892917059 learned_addr learned my addr 172.21.15.44:0/2892917059 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.154+0000 7fa73da00700 1 -- 172.21.15.44:0/2892917059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738105710 con 0x7fa738104bf0 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.154+0000 7fa73da00700 1 --2- 172.21.15.44:0/2892917059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738105000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0098e0 tx=0x7fa72c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879d119d136cc508 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.154+0000 7fa737e00700 1 -- 172.21.15.44:0/2892917059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c00b860 con 0x7fa738104bf0 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.154+0000 7fa737e00700 1 -- 172.21.15.44:0/2892917059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa72c00be90 con 0x7fa738104bf0 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.154+0000 7fa737e00700 1 -- 172.21.15.44:0/2892917059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c0103c0 con 0x7fa738104bf0 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 -- 172.21.15.44:0/2892917059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 msgr2=0x7fa738105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:33.140 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 --2- 172.21.15.44:0/2892917059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738105000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0098e0 tx=0x7fa72c009250 comp rx=0 tx=0).stop 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 -- 172.21.15.44:0/2892917059 shutdown_connections 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 --2- 172.21.15.44:0/2892917059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738105000 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 -- 172.21.15.44:0/2892917059 >> 172.21.15.44:0/2892917059 conn(0x7fa738078320 msgr2=0x7fa738078720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 -- 172.21.15.44:0/2892917059 shutdown_connections 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.155+0000 7fa740400700 1 -- 172.21.15.44:0/2892917059 wait complete. 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.156+0000 7fa740400700 1 Processor -- start 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.156+0000 7fa740400700 1 -- start start 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.157+0000 7fa740400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.157+0000 7fa740400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa738113cb0 con 0x7fa738104bf0 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.157+0000 7fa73da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.157+0000 7fa73da00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738113770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36176/0 (socket says 172.21.15.44:36176) 2024-04-16T11:31:33.145 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.157+0000 7fa73da00700 1 -- 172.21.15.44:0/2983636317 learned_addr learned my addr 172.21.15.44:0/2983636317 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.157+0000 7fa73da00700 1 -- 172.21.15.44:0/2983636317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c0094f0 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.158+0000 7fa73da00700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738113770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0090b0 tx=0x7fa72c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.158+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c010840 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.158+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa72c010e70 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.158+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa738113eb0 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.158+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c024e40 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.158+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa738114a20 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.159+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73805f5f0 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.159+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fa72c024430 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.160+0000 7fa735600700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa720036850 0x7fa720038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.160+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa72c04a810 con 0x7fa738104bf0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.160+0000 7fa73d000700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa720036850 0x7fa720038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.161+0000 7fa73d000700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa720036850 0x7fa720038d00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa724009990 tx=0x7fa724006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:33.146 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.165+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa72c02c930 con 0x7fa738104bf0 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.380+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa73804bed0 con 0x7fa738104bf0 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.387+0000 7fa735600700 1 -- 172.21.15.44:0/2983636317 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa72c02c340 con 0x7fa738104bf0 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa720036850 msgr2=0x7fa720038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa720036850 0x7fa720038d00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa724009990 tx=0x7fa724006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 msgr2=0x7fa738113770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738113770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0090b0 tx=0x7fa72c009250 comp rx=0 tx=0).stop 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 shutdown_connections 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7fa720036850 0x7fa720038d00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 --2- 172.21.15.44:0/2983636317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa738104bf0 0x7fa738113770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.391+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 >> 172.21.15.44:0/2983636317 conn(0x7fa738078320 msgr2=0x7fa7381023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.392+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 shutdown_connections 2024-04-16T11:31:33.147 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:32.392+0000 7fa740400700 1 -- 172.21.15.44:0/2983636317 wait complete. 2024-04-16T11:31:34.778 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.753+0000 7f772c000700 1 Processor -- start 2024-04-16T11:31:34.778 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.753+0000 7f772c000700 1 -- start start 2024-04-16T11:31:34.778 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.753+0000 7f772c000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.753+0000 7f772c000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7724105a40 con 0x7f77240712e0 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.754+0000 7f7729600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.754+0000 7f7729600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36178/0 (socket says 172.21.15.44:36178) 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.754+0000 7f7729600700 1 -- 172.21.15.44:0/2579370845 learned_addr learned my addr 172.21.15.44:0/2579370845 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.754+0000 7f7729600700 1 -- 172.21.15.44:0/2579370845 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7724105b80 con 0x7f77240712e0 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.755+0000 7f7729600700 1 --2- 172.21.15.44:0/2579370845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724105500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f77180098e0 tx=0x7f7718009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ecd49b7d43ce39 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.755+0000 7f7723e00700 1 -- 172.21.15.44:0/2579370845 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f771800b860 con 0x7f77240712e0 2024-04-16T11:31:34.779 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.755+0000 7f7723e00700 1 -- 172.21.15.44:0/2579370845 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f771800be90 con 0x7f77240712e0 2024-04-16T11:31:34.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.755+0000 7f7723e00700 1 -- 172.21.15.44:0/2579370845 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77180103c0 con 0x7f77240712e0 2024-04-16T11:31:34.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 -- 172.21.15.44:0/2579370845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 msgr2=0x7f7724105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:34.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 --2- 172.21.15.44:0/2579370845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724105500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f77180098e0 tx=0x7f7718009250 comp rx=0 tx=0).stop 2024-04-16T11:31:34.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 -- 172.21.15.44:0/2579370845 shutdown_connections 2024-04-16T11:31:34.781 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 --2- 172.21.15.44:0/2579370845 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724105500 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 -- 172.21.15.44:0/2579370845 >> 172.21.15.44:0/2579370845 conn(0x7f77240fcfd0 msgr2=0x7f77240ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 -- 172.21.15.44:0/2579370845 shutdown_connections 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.756+0000 7f772c000700 1 -- 172.21.15.44:0/2579370845 wait complete. 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.757+0000 7f772c000700 1 Processor -- start 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.757+0000 7f772c000700 1 -- start start 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.757+0000 7f772c000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724102e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.758+0000 7f772c000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7724103390 con 0x7f77240712e0 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.758+0000 7f7729600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.758+0000 7f7729600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724102e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36184/0 (socket says 172.21.15.44:36184) 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.758+0000 7f7729600700 1 -- 172.21.15.44:0/2330109027 learned_addr learned my addr 172.21.15.44:0/2330109027 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.758+0000 7f7729600700 1 -- 172.21.15.44:0/2330109027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180094f0 con 0x7f77240712e0 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.759+0000 7f7729600700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724102e50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77180094c0 tx=0x7f7718009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:34.782 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.759+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77180118f0 con 0x7f77240712e0 2024-04-16T11:31:34.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.759+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7718010d60 con 0x7f77240712e0 2024-04-16T11:31:34.783 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.759+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f771801a430 con 0x7f77240712e0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.759+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7724101400 con 0x7f77240712e0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.759+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77241018f0 con 0x7f77240712e0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.760+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f7718024440 con 0x7f77240712e0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.760+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772404bed0 con 0x7f77240712e0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.761+0000 7f7721600700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f770c0368a0 0x7f770c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.761+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f771804a530 con 0x7f77240712e0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.761+0000 7f7728c00700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f770c0368a0 0x7f770c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.762+0000 7f7728c00700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f770c0368a0 0x7f770c038d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7710009990 tx=0x7f7710006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.766+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7718024760 con 0x7f77240712e0 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.983+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7724101d70 con 0x7f77240712e0 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.993+0000 7f7721600700 1 -- 172.21.15.44:0/2330109027 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7718010ed0 con 0x7f77240712e0 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f770c0368a0 msgr2=0x7f770c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f770c0368a0 0x7f770c038d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7710009990 tx=0x7f7710006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 msgr2=0x7f7724102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724102e50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77180094c0 tx=0x7f7718009250 comp rx=0 tx=0).stop 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 shutdown_connections 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f770c0368a0 0x7f770c038d50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 --2- 172.21.15.44:0/2330109027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77240712e0 0x7f7724102e50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.996+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 >> 172.21.15.44:0/2330109027 conn(0x7f77240fcfd0 msgr2=0x7f77240fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.997+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 shutdown_connections 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:33.997+0000 7f772c000700 1 -- 172.21.15.44:0/2330109027 wait complete. 2024-04-16T11:31:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling the dashboard module... 2024-04-16T11:31:35.111 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:34 smithi044 ceph-mon[28731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:35.111 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:34 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2330109027' entity='client.admin' 2024-04-16T11:31:36.439 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:35 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1313346955' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-16T11:31:36.945 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.425+0000 7f7964000700 1 Processor -- start 2024-04-16T11:31:36.945 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.426+0000 7f7964000700 1 -- start start 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.426+0000 7f7964000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c1079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.426+0000 7f7964000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c076f90 con 0x7f795c076b80 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.426+0000 7f7961600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c1079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.427+0000 7f7961600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c1079a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36192/0 (socket says 172.21.15.44:36192) 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.427+0000 7f7961600700 1 -- 172.21.15.44:0/2176959231 learned_addr learned my addr 172.21.15.44:0/2176959231 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.427+0000 7f7961600700 1 -- 172.21.15.44:0/2176959231 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c103390 con 0x7f795c076b80 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.428+0000 7f7961600700 1 --2- 172.21.15.44:0/2176959231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c1079a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79500098e0 tx=0x7f7950009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9c41e964e7e6cc server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.428+0000 7f795be00700 1 -- 172.21.15.44:0/2176959231 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f795000b860 con 0x7f795c076b80 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.428+0000 7f795be00700 1 -- 172.21.15.44:0/2176959231 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f795000be90 con 0x7f795c076b80 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.428+0000 7f795be00700 1 -- 172.21.15.44:0/2176959231 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79500103c0 con 0x7f795c076b80 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 -- 172.21.15.44:0/2176959231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 msgr2=0x7f795c1079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 --2- 172.21.15.44:0/2176959231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c1079a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79500098e0 tx=0x7f7950009250 comp rx=0 tx=0).stop 2024-04-16T11:31:36.946 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 -- 172.21.15.44:0/2176959231 shutdown_connections 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 --2- 172.21.15.44:0/2176959231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c1079a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 -- 172.21.15.44:0/2176959231 >> 172.21.15.44:0/2176959231 conn(0x7f795c0fd010 msgr2=0x7f795c0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 -- 172.21.15.44:0/2176959231 shutdown_connections 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.429+0000 7f7964000700 1 -- 172.21.15.44:0/2176959231 wait complete. 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.430+0000 7f7964000700 1 Processor -- start 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.430+0000 7f7964000700 1 -- start start 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.430+0000 7f7964000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.430+0000 7f7964000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c113d70 con 0x7f795c076b80 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.431+0000 7f7961600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.431+0000 7f7961600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c113830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36200/0 (socket says 172.21.15.44:36200) 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.431+0000 7f7961600700 1 -- 172.21.15.44:0/1313346955 learned_addr learned my addr 172.21.15.44:0/1313346955 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.431+0000 7f7961600700 1 -- 172.21.15.44:0/1313346955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79500094f0 con 0x7f795c076b80 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.431+0000 7f7961600700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c113830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79500105b0 tx=0x7f7950009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.432+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7950010840 con 0x7f795c076b80 2024-04-16T11:31:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.432+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7950010e70 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.432+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c113fd0 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.432+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f795001ae30 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.432+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c114ae0 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.433+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f79500109a0 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.433+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c05f5f0 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.434+0000 7f7959600700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f79440368a0 0x7f7944038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.434+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f795004a110 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.434+0000 7f7960c00700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f79440368a0 0x7f7944038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.435+0000 7f7960c00700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f79440368a0 0x7f7944038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7948009990 tx=0x7f7948006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.439+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f795001a430 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:35.714+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f795c04bed0 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.002+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f795002d3f0 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.005+0000 7f7959600700 1 -- 172.21.15.44:0/1313346955 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f795002c710 con 0x7f795c076b80 2024-04-16T11:31:36.948 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f79440368a0 msgr2=0x7f7944038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f79440368a0 0x7f7944038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7948009990 tx=0x7f7948006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 msgr2=0x7f795c113830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c113830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79500105b0 tx=0x7f7950009250 comp rx=0 tx=0).stop 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 shutdown_connections 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f79440368a0 0x7f7944038d50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 --2- 172.21.15.44:0/1313346955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f795c076b80 0x7f795c113830 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 >> 172.21.15.44:0/1313346955 conn(0x7f795c0fd010 msgr2=0x7f795c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 shutdown_connections 2024-04-16T11:31:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:36.009+0000 7f7964000700 1 -- 172.21.15.44:0/1313346955 wait complete. 2024-04-16T11:31:37.092 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:37 smithi044 ceph-mon[28731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:31:37.092 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:37 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1313346955' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-16T11:31:37.092 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:37 smithi044 ceph-mon[28731]: mgrmap e9: smithi044.sczobv(active, since 32s) 2024-04-16T11:31:38.295 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:38 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/745794075' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T11:31:38.546 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:31:38.546 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-16T11:31:38.546 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T11:31:38.546 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "smithi044.sczobv", 2024-04-16T11:31:38.546 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T11:31:38.546 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.512+0000 7ff085c00700 1 Processor -- start 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.513+0000 7ff085c00700 1 -- start start 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.513+0000 7ff085c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff0800772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.513+0000 7ff085c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800790d0 con 0x7ff080078cc0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.514+0000 7ff07ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff0800772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.514+0000 7ff07ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff0800772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36234/0 (socket says 172.21.15.44:36234) 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.514+0000 7ff07ea00700 1 -- 172.21.15.44:0/2333374193 learned_addr learned my addr 172.21.15.44:0/2333374193 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.514+0000 7ff07ea00700 1 -- 172.21.15.44:0/2333374193 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080077800 con 0x7ff080078cc0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.515+0000 7ff07ea00700 1 --2- 172.21.15.44:0/2333374193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff0800772c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff0700098e0 tx=0x7ff070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8891868271e800da server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.515+0000 7ff07d600700 1 -- 172.21.15.44:0/2333374193 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff07000b860 con 0x7ff080078cc0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.515+0000 7ff07d600700 1 -- 172.21.15.44:0/2333374193 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff07000be90 con 0x7ff080078cc0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.515+0000 7ff07d600700 1 -- 172.21.15.44:0/2333374193 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0700103c0 con 0x7ff080078cc0 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 -- 172.21.15.44:0/2333374193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 msgr2=0x7ff0800772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:38.547 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 --2- 172.21.15.44:0/2333374193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff0800772c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff0700098e0 tx=0x7ff070009250 comp rx=0 tx=0).stop 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 -- 172.21.15.44:0/2333374193 shutdown_connections 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 --2- 172.21.15.44:0/2333374193 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff0800772c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 -- 172.21.15.44:0/2333374193 >> 172.21.15.44:0/2333374193 conn(0x7ff0800fe090 msgr2=0x7ff0801004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 -- 172.21.15.44:0/2333374193 shutdown_connections 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.516+0000 7ff085c00700 1 -- 172.21.15.44:0/2333374193 wait complete. 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.517+0000 7ff085c00700 1 Processor -- start 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.517+0000 7ff085c00700 1 -- start start 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.517+0000 7ff085c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff080117c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.517+0000 7ff085c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801181b0 con 0x7ff080078cc0 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.518+0000 7ff07ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff080117c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.518+0000 7ff07ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff080117c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36250/0 (socket says 172.21.15.44:36250) 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.518+0000 7ff07ea00700 1 -- 172.21.15.44:0/745794075 learned_addr learned my addr 172.21.15.44:0/745794075 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.518+0000 7ff07ea00700 1 -- 172.21.15.44:0/745794075 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700094f0 con 0x7ff080078cc0 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.519+0000 7ff07ea00700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff080117c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff070006fd0 tx=0x7ff070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:38.548 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.519+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff070010630 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.519+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff070010c60 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.519+0000 7ff085c00700 1 -- 172.21.15.44:0/745794075 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0801183b0 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.519+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff07001ace0 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.519+0000 7ff085c00700 1 -- 172.21.15.44:0/745794075 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0801187e0 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.520+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7ff070022420 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.521+0000 7ff076a00700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 0x7ff064038da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.521+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0580052b0 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.521+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff07001e070 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.521+0000 7ff07e000700 1 -- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 msgr2=0x7ff064038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.521+0000 7ff07e000700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 0x7ff064038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.529+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff070020920 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.722+0000 7ff07e000700 1 -- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 msgr2=0x7ff064038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.722+0000 7ff07e000700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 0x7ff064038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.795+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff058005d70 con 0x7ff080078cc0 2024-04-16T11:31:38.549 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.795+0000 7ff076a00700 1 -- 172.21.15.44:0/745794075 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7ff070020330 con 0x7ff080078cc0 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 msgr2=0x7ff064038da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 0x7ff064038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 msgr2=0x7ff080117c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff080117c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff070006fd0 tx=0x7ff070009250 comp rx=0 tx=0).stop 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 shutdown_connections 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7ff0640368f0 0x7ff064038da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 --2- 172.21.15.44:0/745794075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff080078cc0 0x7ff080117c70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 >> 172.21.15.44:0/745794075 conn(0x7ff0800fe090 msgr2=0x7ff08010f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 shutdown_connections 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:37.798+0000 7ff06be00700 1 -- 172.21.15.44:0/745794075 wait complete. 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-04-16T11:31:38.550 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 9... 2024-04-16T11:31:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: Active manager daemon smithi044.sczobv restarted 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: Activating manager daemon smithi044.sczobv 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: osdmap e3: 0 total, 0 up, 0 in 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: mgrmap e10: smithi044.sczobv(active, starting, since 0.0103381s) 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:31:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:44 smithi044 ceph-mon[28731]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:31:46.158 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:45 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:46.158 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:45 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:31:46.158 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:45 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:31:46.158 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:45 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:31:46.159 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:45 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:46.159 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:45 smithi044 ceph-mon[28731]: mgrmap e11: smithi044.sczobv(active, since 1.01242s) 2024-04-16T11:31:46.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-16T11:31:46.973 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-16T11:31:46.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T11:31:46.974 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-16T11:31:46.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.171+0000 7f715ba00700 1 Processor -- start 2024-04-16T11:31:46.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.171+0000 7f715ba00700 1 -- start start 2024-04-16T11:31:46.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.172+0000 7f715ba00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 0x7f7154076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:46.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.172+0000 7f715ba00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154077fb0 con 0x7f7154077ba0 2024-04-16T11:31:46.975 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.172+0000 7f7159000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 0x7f7154076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.172+0000 7f7159000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 0x7f7154076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36262/0 (socket says 172.21.15.44:36262) 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.172+0000 7f7159000700 1 -- 172.21.15.44:0/1691537792 learned_addr learned my addr 172.21.15.44:0/1691537792 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.173+0000 7f7159000700 1 -- 172.21.15.44:0/1691537792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154076690 con 0x7f7154077ba0 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.173+0000 7f7159000700 1 --2- 172.21.15.44:0/1691537792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 0x7f7154076150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f71480098e0 tx=0x7f7148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1527af476cb513 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.173+0000 7f7153400700 1 -- 172.21.15.44:0/1691537792 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714800b860 con 0x7f7154077ba0 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.173+0000 7f7153400700 1 -- 172.21.15.44:0/1691537792 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f714800be90 con 0x7f7154077ba0 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.173+0000 7f7153400700 1 -- 172.21.15.44:0/1691537792 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71480103c0 con 0x7f7154077ba0 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 -- 172.21.15.44:0/1691537792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 msgr2=0x7f7154076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:46.977 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 --2- 172.21.15.44:0/1691537792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 0x7f7154076150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f71480098e0 tx=0x7f7148009250 comp rx=0 tx=0).stop 2024-04-16T11:31:46.980 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 -- 172.21.15.44:0/1691537792 shutdown_connections 2024-04-16T11:31:46.980 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 --2- 172.21.15.44:0/1691537792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7154077ba0 0x7f7154076150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:46.980 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 -- 172.21.15.44:0/1691537792 >> 172.21.15.44:0/1691537792 conn(0x7f71540fe0a0 msgr2=0x7f71541004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:46.980 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 -- 172.21.15.44:0/1691537792 shutdown_connections 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.174+0000 7f715ba00700 1 -- 172.21.15.44:0/1691537792 wait complete. 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.175+0000 7f715ba00700 1 Processor -- start 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.175+0000 7f715ba00700 1 -- start start 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.176+0000 7f715ba00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 0x7f7154113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.176+0000 7f715ba00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154114310 con 0x7f71541139c0 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.176+0000 7f7159000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 0x7f7154113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.176+0000 7f7159000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 0x7f7154113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36268/0 (socket says 172.21.15.44:36268) 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.176+0000 7f7159000700 1 -- 172.21.15.44:0/1708544798 learned_addr learned my addr 172.21.15.44:0/1708544798 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.176+0000 7f7159000700 1 -- 172.21.15.44:0/1708544798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71480094f0 con 0x7f71541139c0 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.177+0000 7f7159000700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 0x7f7154113dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7148011c10 tx=0x7f7148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.177+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148011890 con 0x7f71541139c0 2024-04-16T11:31:46.981 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.177+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f714801f070 con 0x7f71541139c0 2024-04-16T11:31:46.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.177+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154114510 con 0x7f71541139c0 2024-04-16T11:31:46.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.177+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148026990 con 0x7f71541139c0 2024-04-16T11:31:46.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.177+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7154114ef0 con 0x7f71541139c0 2024-04-16T11:31:46.982 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.178+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f7148026af0 con 0x7f71541139c0 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.179+0000 7f7150c00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.179+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 --> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f713c0394b0 con 0x7f713c0368f0 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.179+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7148021070 con 0x7f71541139c0 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.179+0000 7f7153e00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 msgr2=0x7f713c038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.179+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.380+0000 7f7153e00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 msgr2=0x7f713c038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:46.986 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.380+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.781+0000 7f7153e00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 msgr2=0x7f713c038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:39.781+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:40.582+0000 7f7153e00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 msgr2=0x7f713c038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:40.582+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:42.184+0000 7f7153e00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 msgr2=0x7f713c038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2358899774 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:42.184+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:44.885+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 10) v1 ==== 43156+0+0 (secure 0 0 0) 0x7f714801ace0 con 0x7f71541139c0 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:44.885+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 msgr2=0x7f713c038da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:44.885+0000 7f7150c00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.887+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 11) v1 ==== 43283+0+0 (secure 0 0 0) 0x7f714801a440 con 0x7f71541139c0 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.888+0000 7f7150c00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f713c03a3a0 0x7f713c03c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.888+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f713c0394b0 con 0x7f713c03a3a0 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.889+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f713c03a3a0 0x7f713c03c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:46.987 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.891+0000 7f7153e00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f713c03a3a0 0x7f713c03c780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71400039d0 tx=0x7f7140009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.891+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f713c0394b0 con 0x7f713c03a3a0 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.904+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f715404bed0 con 0x7f713c03a3a0 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.905+0000 7f7150c00700 1 -- 172.21.15.44:0/1708544798 <== mgr.14162 v2:172.21.15.44:6800/3690066543 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f715404bed0 con 0x7f713c03a3a0 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.905+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f713c03a3a0 msgr2=0x7f713c03c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.905+0000 7f715ba00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f713c03a3a0 0x7f713c03c780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71400039d0 tx=0x7f7140009350 comp rx=0 tx=0).stop 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.905+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 msgr2=0x7f7154113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.905+0000 7f715ba00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 0x7f7154113dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7148011c10 tx=0x7f7148009250 comp rx=0 tx=0).stop 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 shutdown_connections 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f713c03a3a0 0x7f713c03c780 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71541139c0 0x7f7154113dd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 --2- 172.21.15.44:0/1708544798 >> [v2:172.21.15.44:6800/2358899774,v1:172.21.15.44:6801/2358899774] conn(0x7f713c0368f0 0x7f713c038da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 >> 172.21.15.44:0/1708544798 conn(0x7f71540fe0a0 msgr2=0x7f71540fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 shutdown_connections 2024-04-16T11:31:46.988 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:45.906+0000 7f715ba00700 1 -- 172.21.15.44:0/1708544798 wait complete. 2024-04-16T11:31:46.989 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 9 is available 2024-04-16T11:31:46.989 INFO:teuthology.orchestra.run.smithi044.stdout:Generating a dashboard self-signed certificate... 2024-04-16T11:31:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:46 smithi044 ceph-mon[28731]: [16/Apr/2024:11:31:45] ENGINE Bus STARTING 2024-04-16T11:31:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:46 smithi044 ceph-mon[28731]: [16/Apr/2024:11:31:45] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:31:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:46 smithi044 ceph-mon[28731]: [16/Apr/2024:11:31:45] ENGINE Bus STARTED 2024-04-16T11:31:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:46 smithi044 ceph-mon[28731]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T11:31:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:46 smithi044 ceph-mon[28731]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T11:31:48.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:47 smithi044 ceph-mon[28731]: mgrmap e12: smithi044.sczobv(active, since 2s) 2024-04-16T11:31:48.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:47 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:49.400 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-16T11:31:49.400 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.062+0000 7fea6c000700 1 Processor -- start 2024-04-16T11:31:49.400 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.062+0000 7fea6c000700 1 -- start start 2024-04-16T11:31:49.400 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.062+0000 7fea6c000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:49.400 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.063+0000 7fea6c000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea64103250 con 0x7fea64102900 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.063+0000 7fea69600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.063+0000 7fea69600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49906/0 (socket says 172.21.15.44:49906) 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.063+0000 7fea69600700 1 -- 172.21.15.44:0/2927828818 learned_addr learned my addr 172.21.15.44:0/2927828818 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.064+0000 7fea69600700 1 -- 172.21.15.44:0/2927828818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64103390 con 0x7fea64102900 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.064+0000 7fea69600700 1 --2- 172.21.15.44:0/2927828818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64102d10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fea580098e0 tx=0x7fea58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6aa811f0d5ef885 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.064+0000 7fea63e00700 1 -- 172.21.15.44:0/2927828818 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea5800b860 con 0x7fea64102900 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea63e00700 1 -- 172.21.15.44:0/2927828818 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea5800be90 con 0x7fea64102900 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea63e00700 1 -- 172.21.15.44:0/2927828818 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea580103c0 con 0x7fea64102900 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea6c000700 1 -- 172.21.15.44:0/2927828818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 msgr2=0x7fea64102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea6c000700 1 --2- 172.21.15.44:0/2927828818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64102d10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fea580098e0 tx=0x7fea58009250 comp rx=0 tx=0).stop 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea6c000700 1 -- 172.21.15.44:0/2927828818 shutdown_connections 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea6c000700 1 --2- 172.21.15.44:0/2927828818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64102d10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:49.401 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.065+0000 7fea6c000700 1 -- 172.21.15.44:0/2927828818 >> 172.21.15.44:0/2927828818 conn(0x7fea640fe0a0 msgr2=0x7fea641004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.067+0000 7fea6c000700 1 -- 172.21.15.44:0/2927828818 shutdown_connections 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.067+0000 7fea6c000700 1 -- 172.21.15.44:0/2927828818 wait complete. 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.068+0000 7fea6c000700 1 Processor -- start 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.068+0000 7fea6c000700 1 -- start start 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.068+0000 7fea6c000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64111650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.069+0000 7fea6c000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea64111b90 con 0x7fea64102900 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.069+0000 7fea69600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64111650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.069+0000 7fea69600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64111650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49914/0 (socket says 172.21.15.44:49914) 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.069+0000 7fea69600700 1 -- 172.21.15.44:0/2292559156 learned_addr learned my addr 172.21.15.44:0/2292559156 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.069+0000 7fea69600700 1 -- 172.21.15.44:0/2292559156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580094f0 con 0x7fea64102900 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.070+0000 7fea69600700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64111650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fea58011c10 tx=0x7fea58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.070+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea58010890 con 0x7fea64102900 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.070+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea58010ec0 con 0x7fea64102900 2024-04-16T11:31:49.402 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.070+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea64111d90 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.070+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea580243f0 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.070+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea64112900 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.072+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7fea5801a430 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.071+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6405f5f0 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.072+0000 7fea61600700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fea4c0368f0 0x7fea4c038da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.072+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fea5804a3d0 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.072+0000 7fea68c00700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fea4c0368f0 0x7fea4c038da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.073+0000 7fea68c00700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fea4c0368f0 0x7fea4c038da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea50009990 tx=0x7fea50006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.077+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fea5802b8e0 con 0x7fea64102900 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.303+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fea640ff240 con 0x7fea4c0368f0 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.529+0000 7fea61600700 1 -- 172.21.15.44:0/2292559156 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fea640ff240 con 0x7fea4c0368f0 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fea4c0368f0 msgr2=0x7fea4c038da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fea4c0368f0 0x7fea4c038da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea50009990 tx=0x7fea50006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:49.403 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 msgr2=0x7fea64111650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64111650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fea58011c10 tx=0x7fea58009250 comp rx=0 tx=0).stop 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 shutdown_connections 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fea4c0368f0 0x7fea4c038da0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 --2- 172.21.15.44:0/2292559156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea64102900 0x7fea64111650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.532+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 >> 172.21.15.44:0/2292559156 conn(0x7fea640fe0a0 msgr2=0x7fea640feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.533+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 shutdown_connections 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:48.533+0000 7fea6c000700 1 -- 172.21.15.44:0/2292559156 wait complete. 2024-04-16T11:31:49.404 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial admin user... 2024-04-16T11:31:49.683 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:49 smithi044 ceph-mon[28731]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:49.683 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:49 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:49.683 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:49 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:49.683 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:49 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:51.906 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:51 smithi044 ceph-mon[28731]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:31:51.906 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:51 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:31:51.925 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$HPdipCNQdBghWuQXrZPNAe6MEn35/m.4jEUH46BFN9q1RiVUImn8u", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713267110, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-16T11:31:51.925 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.977+0000 7f1ab3c00700 1 Processor -- start 2024-04-16T11:31:51.925 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.977+0000 7f1ab3c00700 1 -- start start 2024-04-16T11:31:51.925 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.978+0000 7f1ab3c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:51.925 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.978+0000 7f1ab3c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac103250 con 0x7f1aac102900 2024-04-16T11:31:51.926 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.978+0000 7f1ab1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:51.926 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.978+0000 7f1ab1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49930/0 (socket says 172.21.15.44:49930) 2024-04-16T11:31:51.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.978+0000 7f1ab1200700 1 -- 172.21.15.44:0/3331435799 learned_addr learned my addr 172.21.15.44:0/3331435799 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:51.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.979+0000 7f1ab1200700 1 -- 172.21.15.44:0/3331435799 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac103390 con 0x7f1aac102900 2024-04-16T11:31:51.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.979+0000 7f1ab1200700 1 --2- 172.21.15.44:0/3331435799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac102d10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00098e0 tx=0x7f1aa0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f257d6ac0041dc70 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:51.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.979+0000 7f1aab400700 1 -- 172.21.15.44:0/3331435799 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa000b860 con 0x7f1aac102900 2024-04-16T11:31:51.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1aab400700 1 -- 172.21.15.44:0/3331435799 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aa000be90 con 0x7f1aac102900 2024-04-16T11:31:51.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1aab400700 1 -- 172.21.15.44:0/3331435799 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa00103c0 con 0x7f1aac102900 2024-04-16T11:31:51.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1ab3c00700 1 -- 172.21.15.44:0/3331435799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 msgr2=0x7f1aac102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:51.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1ab3c00700 1 --2- 172.21.15.44:0/3331435799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac102d10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00098e0 tx=0x7f1aa0009250 comp rx=0 tx=0).stop 2024-04-16T11:31:51.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1ab3c00700 1 -- 172.21.15.44:0/3331435799 shutdown_connections 2024-04-16T11:31:51.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1ab3c00700 1 --2- 172.21.15.44:0/3331435799 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac102d10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:51.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.980+0000 7f1ab3c00700 1 -- 172.21.15.44:0/3331435799 >> 172.21.15.44:0/3331435799 conn(0x7f1aac0fe0a0 msgr2=0x7f1aac1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:51.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.981+0000 7f1ab3c00700 1 -- 172.21.15.44:0/3331435799 shutdown_connections 2024-04-16T11:31:51.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.981+0000 7f1ab3c00700 1 -- 172.21.15.44:0/3331435799 wait complete. 2024-04-16T11:31:51.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.982+0000 7f1ab3c00700 1 Processor -- start 2024-04-16T11:31:51.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.982+0000 7f1ab3c00700 1 -- start start 2024-04-16T11:31:51.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.982+0000 7f1ab3c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.982+0000 7f1ab3c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac113dd0 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.982+0000 7f1ab1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.983+0000 7f1ab1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49946/0 (socket says 172.21.15.44:49946) 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.983+0000 7f1ab1200700 1 -- 172.21.15.44:0/447104144 learned_addr learned my addr 172.21.15.44:0/447104144 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.983+0000 7f1ab1200700 1 -- 172.21.15.44:0/447104144 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00094f0 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.983+0000 7f1ab1200700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac113890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0006b00 tx=0x7f1aa0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.984+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa0010840 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.984+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aa0010e70 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.984+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aac113fd0 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.984+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa001ae70 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.984+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac114b40 con 0x7f1aac102900 2024-04-16T11:31:51.933 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.985+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f1aa0024480 con 0x7f1aac102900 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.986+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac05f5f0 con 0x7f1aac102900 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.986+0000 7f1aa8c00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f1a940368f0 0x7f1a94038da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.986+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1aa004ac20 con 0x7f1aac102900 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.986+0000 7f1aabe00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f1a940368f0 0x7f1a94038da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.987+0000 7f1aabe00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f1a940368f0 0x7f1a94038da0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009990 tx=0x7f1a98006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:49.995+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1aa002e440 con 0x7f1aac102900 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.269+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1aac05f410 con 0x7f1a940368f0 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.557+0000 7f1aa8c00700 1 -- 172.21.15.44:0/447104144 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1aac05f410 con 0x7f1a940368f0 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.559+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f1a940368f0 msgr2=0x7f1a94038da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.559+0000 7f1ab3c00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f1a940368f0 0x7f1a94038da0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009990 tx=0x7f1a98006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.559+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 msgr2=0x7f1aac113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.559+0000 7f1ab3c00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac113890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0006b00 tx=0x7f1aa0009250 comp rx=0 tx=0).stop 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.560+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 shutdown_connections 2024-04-16T11:31:51.934 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.560+0000 7f1ab3c00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f1a940368f0 0x7f1a94038da0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:51.935 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.560+0000 7f1ab3c00700 1 --2- 172.21.15.44:0/447104144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1aac102900 0x7f1aac113890 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:51.935 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.560+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 >> 172.21.15.44:0/447104144 conn(0x7f1aac0fe0a0 msgr2=0x7f1aac0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:51.935 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.560+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 shutdown_connections 2024-04-16T11:31:51.935 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:50.560+0000 7f1ab3c00700 1 -- 172.21.15.44:0/447104144 wait complete. 2024-04-16T11:31:51.936 INFO:teuthology.orchestra.run.smithi044.stdout:Fetching dashboard port number... 2024-04-16T11:31:52.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:52 smithi044 ceph-mon[28731]: mgrmap e13: smithi044.sczobv(active, since 6s) 2024-04-16T11:31:53.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 8443 2024-04-16T11:31:53.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.527+0000 7fb7cc600700 1 Processor -- start 2024-04-16T11:31:53.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.527+0000 7fb7cc600700 1 -- start start 2024-04-16T11:31:53.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.528+0000 7fb7cc600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c40772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.528+0000 7fb7cc600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40790d0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.529+0000 7fb7c9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c40772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.529+0000 7fb7c9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c40772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46120/0 (socket says 172.21.15.44:46120) 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.529+0000 7fb7c9c00700 1 -- 172.21.15.44:0/2639584887 learned_addr learned my addr 172.21.15.44:0/2639584887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.529+0000 7fb7c9c00700 1 -- 172.21.15.44:0/2639584887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4077800 con 0x7fb7c4078cc0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.530+0000 7fb7c9c00700 1 --2- 172.21.15.44:0/2639584887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c40772c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80098e0 tx=0x7fb7b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5743a8e2dad2aa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.530+0000 7fb7c3e00700 1 -- 172.21.15.44:0/2639584887 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b800b860 con 0x7fb7c4078cc0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.530+0000 7fb7c3e00700 1 -- 172.21.15.44:0/2639584887 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7b800be90 con 0x7fb7c4078cc0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.530+0000 7fb7c3e00700 1 -- 172.21.15.44:0/2639584887 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b80103c0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 -- 172.21.15.44:0/2639584887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 msgr2=0x7fb7c40772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 --2- 172.21.15.44:0/2639584887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c40772c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80098e0 tx=0x7fb7b8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:53.460 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 -- 172.21.15.44:0/2639584887 shutdown_connections 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 --2- 172.21.15.44:0/2639584887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c40772c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 -- 172.21.15.44:0/2639584887 >> 172.21.15.44:0/2639584887 conn(0x7fb7c40fe0b0 msgr2=0x7fb7c41004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 -- 172.21.15.44:0/2639584887 shutdown_connections 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.531+0000 7fb7cc600700 1 -- 172.21.15.44:0/2639584887 wait complete. 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.532+0000 7fb7cc600700 1 Processor -- start 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.533+0000 7fb7cc600700 1 -- start start 2024-04-16T11:31:53.461 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.533+0000 7fb7cc600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c41138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:53.464 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.533+0000 7fb7cc600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c4113e00 con 0x7fb7c4078cc0 2024-04-16T11:31:53.464 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.533+0000 7fb7c9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c41138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:53.464 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.533+0000 7fb7c9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c41138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46124/0 (socket says 172.21.15.44:46124) 2024-04-16T11:31:53.464 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.533+0000 7fb7c9c00700 1 -- 172.21.15.44:0/2354433190 learned_addr learned my addr 172.21.15.44:0/2354433190 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.534+0000 7fb7c9c00700 1 -- 172.21.15.44:0/2354433190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b80094f0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.534+0000 7fb7c9c00700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c41138c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8011d50 tx=0x7fb7b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.534+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b8010630 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.534+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7b8010c60 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.534+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c4114000 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.534+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b801ace0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.535+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c4114b70 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.536+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c405f5f0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.536+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fb7b8010790 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.537+0000 7fb7c1600700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb7ac036990 0x7fb7ac038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.537+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb7b804a1c0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.537+0000 7fb7c9200700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb7ac036990 0x7fb7ac038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:53.465 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.538+0000 7fb7c9200700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb7ac036990 0x7fb7ac038e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009990 tx=0x7fb7b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.542+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb7b801a560 con 0x7fb7c4078cc0 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.758+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb7c404bed0 con 0x7fb7c4078cc0 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.759+0000 7fb7c1600700 1 -- 172.21.15.44:0/2354433190 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb7b802a070 con 0x7fb7c4078cc0 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb7ac036990 msgr2=0x7fb7ac038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb7ac036990 0x7fb7ac038e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009990 tx=0x7fb7b0006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 msgr2=0x7fb7c41138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c41138c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8011d50 tx=0x7fb7b8009250 comp rx=0 tx=0).stop 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 shutdown_connections 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb7ac036990 0x7fb7ac038e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 --2- 172.21.15.44:0/2354433190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c4078cc0 0x7fb7c41138c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 >> 172.21.15.44:0/2354433190 conn(0x7fb7c40fe0b0 msgr2=0x7fb7c40fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.762+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 shutdown_connections 2024-04-16T11:31:53.466 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:52.763+0000 7fb7cc600700 1 -- 172.21.15.44:0/2354433190 wait complete. 2024-04-16T11:31:53.498 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T11:31:53.499 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-16T11:31:53.523 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T11:31:53.523 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-16T11:31:53.524 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-16T11:31:53.524 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-16T11:31:53.528 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph Dashboard is now available at: 2024-04-16T11:31:53.528 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:53.529 INFO:teuthology.orchestra.run.smithi044.stdout: URL: https://smithi044.front.sepia.ceph.com:8443/ 2024-04-16T11:31:53.529 INFO:teuthology.orchestra.run.smithi044.stdout: User: admin 2024-04-16T11:31:53.529 INFO:teuthology.orchestra.run.smithi044.stdout: Password: amph8mwgkn 2024-04-16T11:31:53.529 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:53.529 INFO:teuthology.orchestra.run.smithi044.stdout:Saving cluster configuration to /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config directory 2024-04-16T11:31:53.531 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling autotune for osd_memory_target 2024-04-16T11:31:53.696 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:53 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2354433190' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-16T11:31:55.094 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.176+0000 7f19c7600700 1 Processor -- start 2024-04-16T11:31:55.094 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.177+0000 7f19c7600700 1 -- start start 2024-04-16T11:31:55.094 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.177+0000 7f19c7600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c007ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:55.094 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.177+0000 7f19c7600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c007d420 con 0x7f19c007ca40 2024-04-16T11:31:55.094 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.178+0000 7f19c4c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c007ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:55.094 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.178+0000 7f19c4c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c007ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46136/0 (socket says 172.21.15.44:46136) 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.178+0000 7f19c4c00700 1 -- 172.21.15.44:0/243281217 learned_addr learned my addr 172.21.15.44:0/243281217 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.179+0000 7f19c4c00700 1 -- 172.21.15.44:0/243281217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c007d560 con 0x7f19c007ca40 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.179+0000 7f19c4c00700 1 --2- 172.21.15.44:0/243281217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c007ce50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19b40098e0 tx=0x7f19b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187c32c521e2185c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.179+0000 7f19bf400700 1 -- 172.21.15.44:0/243281217 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b400b860 con 0x7f19c007ca40 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.179+0000 7f19bf400700 1 -- 172.21.15.44:0/243281217 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19b400be90 con 0x7f19c007ca40 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.179+0000 7f19bf400700 1 -- 172.21.15.44:0/243281217 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b40103c0 con 0x7f19c007ca40 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 -- 172.21.15.44:0/243281217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 msgr2=0x7f19c007ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 --2- 172.21.15.44:0/243281217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c007ce50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19b40098e0 tx=0x7f19b4009250 comp rx=0 tx=0).stop 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 -- 172.21.15.44:0/243281217 shutdown_connections 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 --2- 172.21.15.44:0/243281217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c007ce50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 -- 172.21.15.44:0/243281217 >> 172.21.15.44:0/243281217 conn(0x7f19c0078350 msgr2=0x7f19c007a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 -- 172.21.15.44:0/243281217 shutdown_connections 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.180+0000 7f19c7600700 1 -- 172.21.15.44:0/243281217 wait complete. 2024-04-16T11:31:55.095 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.181+0000 7f19c7600700 1 Processor -- start 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.181+0000 7f19c7600700 1 -- start start 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.182+0000 7f19c7600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c0117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.182+0000 7f19c7600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0118150 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.182+0000 7f19c4c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c0117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.182+0000 7f19c4c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c0117c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46140/0 (socket says 172.21.15.44:46140) 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.182+0000 7f19c4c00700 1 -- 172.21.15.44:0/3290837461 learned_addr learned my addr 172.21.15.44:0/3290837461 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.182+0000 7f19c4c00700 1 -- 172.21.15.44:0/3290837461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b40094f0 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.183+0000 7f19c4c00700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c0117c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f19b4011c10 tx=0x7f19b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.183+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b40118f0 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.183+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19b4010d60 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.183+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c0118350 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.183+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b40243f0 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.183+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c0118ec0 con 0x7f19c007ca40 2024-04-16T11:31:55.096 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.184+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f19b401a7a0 con 0x7f19c007ca40 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.185+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c005f5f0 con 0x7f19c007ca40 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.185+0000 7f19bcc00700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f19a80369e0 0x7f19a8038e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.185+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f19b404a850 con 0x7f19c007ca40 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.185+0000 7f19bfe00700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f19a80369e0 0x7f19a8038e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.186+0000 7f19bfe00700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f19a80369e0 0x7f19a8038e90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f19ac009990 tx=0x7f19ac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.191+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f19b4024550 con 0x7f19c007ca40 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.406+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f19c004bed0 con 0x7f19c007ca40 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.408+0000 7f19bcc00700 1 -- 172.21.15.44:0/3290837461 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f19b4010ed0 con 0x7f19c007ca40 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.410+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f19a80369e0 msgr2=0x7f19a8038e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.410+0000 7f19c7600700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f19a80369e0 0x7f19a8038e90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f19ac009990 tx=0x7f19ac006e40 comp rx=0 tx=0).stop 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.410+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 msgr2=0x7f19c0117c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.410+0000 7f19c7600700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c0117c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f19b4011c10 tx=0x7f19b4009250 comp rx=0 tx=0).stop 2024-04-16T11:31:55.099 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.410+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 shutdown_connections 2024-04-16T11:31:55.100 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.411+0000 7f19c7600700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f19a80369e0 0x7f19a8038e90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:55.100 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.411+0000 7f19c7600700 1 --2- 172.21.15.44:0/3290837461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19c007ca40 0x7f19c0117c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:55.100 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.411+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 >> 172.21.15.44:0/3290837461 conn(0x7f19c0078350 msgr2=0x7f19c0079da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:55.100 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.411+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 shutdown_connections 2024-04-16T11:31:55.100 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:54.411+0000 7f19c7600700 1 -- 172.21.15.44:0/3290837461 wait complete. 2024-04-16T11:31:56.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.749+0000 7fb0a5400700 1 Processor -- start 2024-04-16T11:31:56.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.749+0000 7fb0a5400700 1 -- start start 2024-04-16T11:31:56.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.749+0000 7fb0a5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:56.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.749+0000 7fb0a5400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0105600 con 0x7fb0a0104c20 2024-04-16T11:31:56.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.750+0000 7fb09e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:56.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.750+0000 7fb09e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46150/0 (socket says 172.21.15.44:46150) 2024-04-16T11:31:56.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.750+0000 7fb09e800700 1 -- 172.21.15.44:0/1061948338 learned_addr learned my addr 172.21.15.44:0/1061948338 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:56.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.750+0000 7fb09e800700 1 -- 172.21.15.44:0/1061948338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0105740 con 0x7fb0a0104c20 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.751+0000 7fb09e800700 1 --2- 172.21.15.44:0/1061948338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0105030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098e0 tx=0x7fb090009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c89172786a6b47 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.751+0000 7fb09de00700 1 -- 172.21.15.44:0/1061948338 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb09000b860 con 0x7fb0a0104c20 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.751+0000 7fb09de00700 1 -- 172.21.15.44:0/1061948338 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb09000be90 con 0x7fb0a0104c20 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.751+0000 7fb09de00700 1 -- 172.21.15.44:0/1061948338 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0900103c0 con 0x7fb0a0104c20 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 -- 172.21.15.44:0/1061948338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 msgr2=0x7fb0a0105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 --2- 172.21.15.44:0/1061948338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0105030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098e0 tx=0x7fb090009250 comp rx=0 tx=0).stop 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 -- 172.21.15.44:0/1061948338 shutdown_connections 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 --2- 172.21.15.44:0/1061948338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0105030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 -- 172.21.15.44:0/1061948338 >> 172.21.15.44:0/1061948338 conn(0x7fb0a0078350 msgr2=0x7fb0a0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 -- 172.21.15.44:0/1061948338 shutdown_connections 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.752+0000 7fb0a5400700 1 -- 172.21.15.44:0/1061948338 wait complete. 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.753+0000 7fb0a5400700 1 Processor -- start 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.753+0000 7fb0a5400700 1 -- start start 2024-04-16T11:31:56.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.753+0000 7fb0a5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:56.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.753+0000 7fb0a5400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0113d60 con 0x7fb0a0104c20 2024-04-16T11:31:56.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.754+0000 7fb09e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:56.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.754+0000 7fb09e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46166/0 (socket says 172.21.15.44:46166) 2024-04-16T11:31:56.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.754+0000 7fb09e800700 1 -- 172.21.15.44:0/3243764480 learned_addr learned my addr 172.21.15.44:0/3243764480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:31:56.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.754+0000 7fb09e800700 1 -- 172.21.15.44:0/3243764480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0900094f0 con 0x7fb0a0104c20 2024-04-16T11:31:56.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.754+0000 7fb09e800700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0113820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb090006b00 tx=0x7fb090009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:56.767 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.755+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb090010840 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.755+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb090010e70 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.755+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a0113f60 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.755+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb09001ae70 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.755+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a0114ad0 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.756+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a005f5f0 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.756+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fb09001a780 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.757+0000 7fb097400700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb084036990 0x7fb084038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.757+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb09004ae30 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.757+0000 7fb095e00700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb084036990 0x7fb084038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.758+0000 7fb095e00700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb084036990 0x7fb084038e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb08800ad90 tx=0x7fb088009410 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:55.762+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb09004f070 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.092+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb0a004bed0 con 0x7fb0a0104c20 2024-04-16T11:31:56.768 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.096+0000 7fb097400700 1 -- 172.21.15.44:0/3243764480 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb09001a430 con 0x7fb0a0104c20 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.098+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb084036990 msgr2=0x7fb084038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.098+0000 7fb0a5400700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb084036990 0x7fb084038e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb08800ad90 tx=0x7fb088009410 comp rx=0 tx=0).stop 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.098+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 msgr2=0x7fb0a0113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.098+0000 7fb0a5400700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0113820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb090006b00 tx=0x7fb090009250 comp rx=0 tx=0).stop 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.099+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 shutdown_connections 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.099+0000 7fb0a5400700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb084036990 0x7fb084038e40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.099+0000 7fb0a5400700 1 --2- 172.21.15.44:0/3243764480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0a0104c20 0x7fb0a0113820 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.099+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 >> 172.21.15.44:0/3243764480 conn(0x7fb0a0078350 msgr2=0x7fb0a0102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.099+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 shutdown_connections 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-16T11:31:56.099+0000 7fb0a5400700 1 -- 172.21.15.44:0/3243764480 wait complete. 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.769 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout:Or, if you are only running a single cluster on this host: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: ceph telemetry on 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout:For more information see: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-16T11:31:56.770 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:31:56.771 INFO:teuthology.orchestra.run.smithi044.stdout:Bootstrap complete. 2024-04-16T11:31:56.884 INFO:tasks.cephadm:Fetching config... 2024-04-16T11:31:56.885 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:31:56.885 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-16T11:31:56.935 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-16T11:31:56.935 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:31:56.935 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-16T11:31:57.022 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-16T11:31:57.023 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:31:57.023 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/keyring of=/dev/stdout 2024-04-16T11:31:57.155 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-16T11:31:57.155 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:31:57.155 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-16T11:31:57.210 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-16T11:31:57.210 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+RBEvnKbLvMm+wSwSX90n8BwDW4FZEw4F8db7FtqTsjzS1+L5Lv82c4nXK6d9tDhtVoaqCJNVlZ4HRYCATAqPnXeGd5Rxi091osSoF40Cu0iKbG7aajiVCpNF4/hzV6PNAZNsauDalianHQGD7gsvC9Cs69IvyocSVKV1u9mmROjsZl8vG/WmLv9I6pRmN/iq31McYdXEj0vu4Ch91y+dt0BqhyaLJJT8pJCv055uSI7D1r84Czqdft7szLYJNbpkZmNRQ+5pD68SJ29sprc5wS153N500CsiLZLwvBpNBlxOh1OHIIM/N+GIjnpzFUNYFw9XHHg35FgAH1Sn+lEWOYIh5zPeS2WzcQ3rBxJ7O09WGCWG+h2d07T2O4DFxQMmSbubVQrkxTKKfrBktbPATqbfPRJzrhjrDw+npQ7biMEGNSt4Gvmm8lJvnarr919FJFRd+gN066YVIoZNeQ5IVphg8soBJAMYSMxIsMeituBeX39JsotaBDlVpxp+UuM= ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T11:31:57.313 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:31:57 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3243764480' entity='client.admin' 2024-04-16T11:31:57.316 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+RBEvnKbLvMm+wSwSX90n8BwDW4FZEw4F8db7FtqTsjzS1+L5Lv82c4nXK6d9tDhtVoaqCJNVlZ4HRYCATAqPnXeGd5Rxi091osSoF40Cu0iKbG7aajiVCpNF4/hzV6PNAZNsauDalianHQGD7gsvC9Cs69IvyocSVKV1u9mmROjsZl8vG/WmLv9I6pRmN/iq31McYdXEj0vu4Ch91y+dt0BqhyaLJJT8pJCv055uSI7D1r84Czqdft7szLYJNbpkZmNRQ+5pD68SJ29sprc5wS153N500CsiLZLwvBpNBlxOh1OHIIM/N+GIjnpzFUNYFw9XHHg35FgAH1Sn+lEWOYIh5zPeS2WzcQ3rBxJ7O09WGCWG+h2d07T2O4DFxQMmSbubVQrkxTKKfrBktbPATqbfPRJzrhjrDw+npQ7biMEGNSt4Gvmm8lJvnarr919FJFRd+gN066YVIoZNeQ5IVphg8soBJAMYSMxIsMeituBeX39JsotaBDlVpxp+UuM= ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:31:57.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+RBEvnKbLvMm+wSwSX90n8BwDW4FZEw4F8db7FtqTsjzS1+L5Lv82c4nXK6d9tDhtVoaqCJNVlZ4HRYCATAqPnXeGd5Rxi091osSoF40Cu0iKbG7aajiVCpNF4/hzV6PNAZNsauDalianHQGD7gsvC9Cs69IvyocSVKV1u9mmROjsZl8vG/WmLv9I6pRmN/iq31McYdXEj0vu4Ch91y+dt0BqhyaLJJT8pJCv055uSI7D1r84Czqdft7szLYJNbpkZmNRQ+5pD68SJ29sprc5wS153N500CsiLZLwvBpNBlxOh1OHIIM/N+GIjnpzFUNYFw9XHHg35FgAH1Sn+lEWOYIh5zPeS2WzcQ3rBxJ7O09WGCWG+h2d07T2O4DFxQMmSbubVQrkxTKKfrBktbPATqbfPRJzrhjrDw+npQ7biMEGNSt4Gvmm8lJvnarr919FJFRd+gN066YVIoZNeQ5IVphg8soBJAMYSMxIsMeituBeX39JsotaBDlVpxp+UuM= ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T11:31:57.511 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+RBEvnKbLvMm+wSwSX90n8BwDW4FZEw4F8db7FtqTsjzS1+L5Lv82c4nXK6d9tDhtVoaqCJNVlZ4HRYCATAqPnXeGd5Rxi091osSoF40Cu0iKbG7aajiVCpNF4/hzV6PNAZNsauDalianHQGD7gsvC9Cs69IvyocSVKV1u9mmROjsZl8vG/WmLv9I6pRmN/iq31McYdXEj0vu4Ch91y+dt0BqhyaLJJT8pJCv055uSI7D1r84Czqdft7szLYJNbpkZmNRQ+5pD68SJ29sprc5wS153N500CsiLZLwvBpNBlxOh1OHIIM/N+GIjnpzFUNYFw9XHHg35FgAH1Sn+lEWOYIh5zPeS2WzcQ3rBxJ7O09WGCWG+h2d07T2O4DFxQMmSbubVQrkxTKKfrBktbPATqbfPRJzrhjrDw+npQ7biMEGNSt4Gvmm8lJvnarr919FJFRd+gN066YVIoZNeQ5IVphg8soBJAMYSMxIsMeituBeX39JsotaBDlVpxp+UuM= ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:31:57.583 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-16T11:31:58.038 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 -- 172.21.15.44:0/1240626085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 msgr2=0x7f8f1c0fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 --2- 172.21.15.44:0/1240626085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c0fce80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f100048c0 tx=0x7f8f10009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 -- 172.21.15.44:0/1240626085 shutdown_connections 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 --2- 172.21.15.44:0/1240626085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c0fce80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 -- 172.21.15.44:0/1240626085 >> 172.21.15.44:0/1240626085 conn(0x7f8f1c0f81d0 msgr2=0x7f8f1c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 -- 172.21.15.44:0/1240626085 shutdown_connections 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.820+0000 7f8f24800700 1 -- 172.21.15.44:0/1240626085 wait complete. 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.821+0000 7f8f24800700 1 Processor -- start 2024-04-16T11:32:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.821+0000 7f8f24800700 1 -- start start 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.822+0000 7f8f24800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.822+0000 7f8f24800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0722a0 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.822+0000 7f8f21e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.822+0000 7f8f21e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46192/0 (socket says 172.21.15.44:46192) 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.822+0000 7f8f21e00700 1 -- 172.21.15.44:0/1654253068 learned_addr learned my addr 172.21.15.44:0/1654253068 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f21e00700 1 -- 172.21.15.44:0/1654253068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10005040 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f21e00700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c071d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009680 tx=0x7f8f10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f1000b5c0 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f1000be70 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f100135a0 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c070370 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.823+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c070860 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.824+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f8f10013810 con 0x7f8f1c0fca70 2024-04-16T11:32:00.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.824+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c05f5f0 con 0x7f8f1c0fca70 2024-04-16T11:32:00.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.825+0000 7f8f16000700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f8f040368d0 0x7f8f04038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:00.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.825+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8f100493f0 con 0x7f8f1c0fca70 2024-04-16T11:32:00.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.825+0000 7f8f21400700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f8f040368d0 0x7f8f04038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:00.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.826+0000 7f8f21400700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f8f040368d0 0x7f8f04038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009990 tx=0x7f8f0c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:00.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:00.831+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8f10004d70 con 0x7f8f1c0fca70 2024-04-16T11:32:01.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.050+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f8f1c04bed0 con 0x7f8f1c0fca70 2024-04-16T11:32:01.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.057+0000 7f8f16000700 1 -- 172.21.15.44:0/1654253068 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8f10021070 con 0x7f8f1c0fca70 2024-04-16T11:32:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.060+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f8f040368d0 msgr2=0x7f8f04038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.060+0000 7f8f24800700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f8f040368d0 0x7f8f04038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009990 tx=0x7f8f0c006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.060+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 msgr2=0x7f8f1c071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.060+0000 7f8f24800700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c071d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009680 tx=0x7f8f10009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:01.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.061+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 shutdown_connections 2024-04-16T11:32:01.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.061+0000 7f8f24800700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f8f040368d0 0x7f8f04038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:01.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.061+0000 7f8f24800700 1 --2- 172.21.15.44:0/1654253068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f1c0fca70 0x7f8f1c071d60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:01.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.061+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 >> 172.21.15.44:0/1654253068 conn(0x7f8f1c0f81d0 msgr2=0x7f8f1c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:01.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.061+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 shutdown_connections 2024-04-16T11:32:01.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:01.061+0000 7f8f24800700 1 -- 172.21.15.44:0/1654253068 wait complete. 2024-04-16T11:32:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: Deploying daemon crash.smithi044 on smithi044 2024-04-16T11:32:01.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:01 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1654253068' entity='client.admin' 2024-04-16T11:32:01.842 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-16T11:32:01.845 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-16T11:32:02.908 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:32:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 -- 172.21.15.44:0/2823457653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 msgr2=0x7fdadc0feed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 --2- 172.21.15.44:0/2823457653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc0feed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0048c0 tx=0x7fdacc009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 -- 172.21.15.44:0/2823457653 shutdown_connections 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 --2- 172.21.15.44:0/2823457653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc0feed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 -- 172.21.15.44:0/2823457653 >> 172.21.15.44:0/2823457653 conn(0x7fdadc0fa5d0 msgr2=0x7fdadc0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 -- 172.21.15.44:0/2823457653 shutdown_connections 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.632+0000 7fdae0a00700 1 -- 172.21.15.44:0/2823457653 wait complete. 2024-04-16T11:32:05.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.633+0000 7fdae0a00700 1 Processor -- start 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.633+0000 7fdae0a00700 1 -- start start 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.634+0000 7fdae0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc10f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.634+0000 7fdae0a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc10f850 con 0x7fdadc0feac0 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.634+0000 7fdadb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc10f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.634+0000 7fdadb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc10f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43092/0 (socket says 172.21.15.44:43092) 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.634+0000 7fdadb400700 1 -- 172.21.15.44:0/713568787 learned_addr learned my addr 172.21.15.44:0/713568787 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.634+0000 7fdadb400700 1 -- 172.21.15.44:0/713568787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc005040 con 0x7fdadc0feac0 2024-04-16T11:32:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.635+0000 7fdadb400700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc10f310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009680 tx=0x7fdacc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.635+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdacc00b5c0 con 0x7fdadc0feac0 2024-04-16T11:32:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.635+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdacc00be70 con 0x7fdadc0feac0 2024-04-16T11:32:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.635+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdacc0135a0 con 0x7fdadc0feac0 2024-04-16T11:32:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.635+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc10fa50 con 0x7fdadc0feac0 2024-04-16T11:32:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.635+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdadc1105c0 con 0x7fdadc0feac0 2024-04-16T11:32:05.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.636+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fdacc004d30 con 0x7fdadc0feac0 2024-04-16T11:32:05.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.636+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc104fd0 con 0x7fdadc0feac0 2024-04-16T11:32:05.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.637+0000 7fdad3400700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fdac0036880 0x7fdac0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:05.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.637+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdacc049460 con 0x7fdadc0feac0 2024-04-16T11:32:05.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.637+0000 7fdadaa00700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fdac0036880 0x7fdac0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:05.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.638+0000 7fdadaa00700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fdac0036880 0x7fdac0038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009990 tx=0x7fdac8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:05.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.642+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdacc011960 con 0x7fdadc0feac0 2024-04-16T11:32:05.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.898+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fdadc104870 con 0x7fdac0036880 2024-04-16T11:32:05.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.903+0000 7fdad3400700 1 -- 172.21.15.44:0/713568787 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdadc104870 con 0x7fdac0036880 2024-04-16T11:32:05.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fdac0036880 msgr2=0x7fdac0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:05.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fdac0036880 0x7fdac0038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009990 tx=0x7fdac8006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:05.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 msgr2=0x7fdadc10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:05.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc10f310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009680 tx=0x7fdacc009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:05.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 shutdown_connections 2024-04-16T11:32:05.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fdac0036880 0x7fdac0038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:05.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 --2- 172.21.15.44:0/713568787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdadc0feac0 0x7fdadc10f310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:05.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 >> 172.21.15.44:0/713568787 conn(0x7fdadc0fa5d0 msgr2=0x7fdadc0fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:05.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 shutdown_connections 2024-04-16T11:32:05.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:05.906+0000 7fdae0a00700 1 -- 172.21.15.44:0/713568787 wait complete. 2024-04-16T11:32:05.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:05 smithi044 ceph-mon[28731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:05.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:05 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:06.788 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi045 2024-04-16T11:32:06.788 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:32:06.788 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/etc/ceph/ceph.conf 2024-04-16T11:32:06.848 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:32:06.849 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:32:06.944 INFO:tasks.cephadm:Adding host smithi045 to orchestrator... 2024-04-16T11:32:06.945 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch host add smithi045 2024-04-16T11:32:06.996 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:06 smithi044 ceph-mon[28731]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:32:07.511 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:32:08.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:07 smithi044 ceph-mon[28731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:08.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:07 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:08.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:07 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:08.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:07 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:08.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:07 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:09.109 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:08 smithi044 ceph-mon[28731]: Deploying daemon node-exporter.smithi044 on smithi044 2024-04-16T11:32:09.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 -- 172.21.15.44:0/456036381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 msgr2=0x7fb85c0fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:09.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 --2- 172.21.15.44:0/456036381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c0fce80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb84c0048c0 tx=0x7fb84c009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 -- 172.21.15.44:0/456036381 shutdown_connections 2024-04-16T11:32:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 --2- 172.21.15.44:0/456036381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c0fce80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 -- 172.21.15.44:0/456036381 >> 172.21.15.44:0/456036381 conn(0x7fb85c0f81d0 msgr2=0x7fb85c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 -- 172.21.15.44:0/456036381 shutdown_connections 2024-04-16T11:32:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.833+0000 7fb863200700 1 -- 172.21.15.44:0/456036381 wait complete. 2024-04-16T11:32:09.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.834+0000 7fb863200700 1 Processor -- start 2024-04-16T11:32:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.835+0000 7fb863200700 1 -- start start 2024-04-16T11:32:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.835+0000 7fb863200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.835+0000 7fb863200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0722a0 con 0x7fb85c0fca70 2024-04-16T11:32:09.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.835+0000 7fb85be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.835+0000 7fb85be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43102/0 (socket says 172.21.15.44:43102) 2024-04-16T11:32:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.835+0000 7fb85be00700 1 -- 172.21.15.44:0/2286060772 learned_addr learned my addr 172.21.15.44:0/2286060772 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:32:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.836+0000 7fb85be00700 1 -- 172.21.15.44:0/2286060772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c005040 con 0x7fb85c0fca70 2024-04-16T11:32:09.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.836+0000 7fb85be00700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c071d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009680 tx=0x7fb84c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.836+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb84c00b5c0 con 0x7fb85c0fca70 2024-04-16T11:32:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.837+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb84c00be70 con 0x7fb85c0fca70 2024-04-16T11:32:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.837+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb84c0135a0 con 0x7fb85c0fca70 2024-04-16T11:32:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.837+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c070370 con 0x7fb85c0fca70 2024-04-16T11:32:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.837+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c070860 con 0x7fb85c0fca70 2024-04-16T11:32:09.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.838+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c05f5f0 con 0x7fb85c0fca70 2024-04-16T11:32:09.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.838+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fb84c01c430 con 0x7fb85c0fca70 2024-04-16T11:32:09.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.839+0000 7fb853e00700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb8400368d0 0x7fb840038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:09.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.839+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb84c049450 con 0x7fb85c0fca70 2024-04-16T11:32:09.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.839+0000 7fb85b400700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb8400368d0 0x7fb840038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:09.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.840+0000 7fb85b400700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb8400368d0 0x7fb840038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb848009990 tx=0x7fb848006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:09.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:09.844+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb84c004d30 con 0x7fb85c0fca70 2024-04-16T11:32:10.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:09 smithi044 ceph-mon[28731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:10.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:09 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:10.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:10.105+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi045", "target": ["mon-mgr", ""]}) v1 -- 0x7fb85c1064c0 con 0x7fb8400368d0 2024-04-16T11:32:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:10 smithi044 ceph-mon[28731]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:32:12.324 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:11 smithi044 ceph-mon[28731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:13.265 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:12 smithi044 ceph-mon[28731]: Deploying cephadm binary to smithi045 2024-04-16T11:32:14.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:13 smithi044 ceph-mon[28731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.770+0000 7fb853e00700 1 -- 172.21.15.44:0/2286060772 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb85c1064c0 con 0x7fb8400368d0 2024-04-16T11:32:14.770 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi045' with addr '172.21.15.45' 2024-04-16T11:32:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.773+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb8400368d0 msgr2=0x7fb840038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.773+0000 7fb863200700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb8400368d0 0x7fb840038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb848009990 tx=0x7fb848006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.773+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 msgr2=0x7fb85c071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.773+0000 7fb863200700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c071d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009680 tx=0x7fb84c009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.774+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 shutdown_connections 2024-04-16T11:32:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.774+0000 7fb863200700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fb8400368d0 0x7fb840038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.774+0000 7fb863200700 1 --2- 172.21.15.44:0/2286060772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb85c0fca70 0x7fb85c071d60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.774+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 >> 172.21.15.44:0/2286060772 conn(0x7fb85c0f81d0 msgr2=0x7fb85c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.774+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 shutdown_connections 2024-04-16T11:32:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:14.774+0000 7fb863200700 1 -- 172.21.15.44:0/2286060772 wait complete. 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: Deploying daemon alertmanager.smithi044 on smithi044 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: Added host smithi045 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:15.435 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:15 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:15.495 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-16T11:32:15.949 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:32:18.111 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:17 smithi044 ceph-mon[28731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:18.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 -- 172.21.15.44:0/3274834933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 msgr2=0x7f4cb4070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:18.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 --2- 172.21.15.44:0/3274834933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb4070330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80048c0 tx=0x7f4ca8009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 -- 172.21.15.44:0/3274834933 shutdown_connections 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 --2- 172.21.15.44:0/3274834933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb4070330 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 -- 172.21.15.44:0/3274834933 >> 172.21.15.44:0/3274834933 conn(0x7f4cb40f8010 msgr2=0x7f4cb40fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 -- 172.21.15.44:0/3274834933 shutdown_connections 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.738+0000 7f4cbac00700 1 -- 172.21.15.44:0/3274834933 wait complete. 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.739+0000 7f4cbac00700 1 Processor -- start 2024-04-16T11:32:18.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.739+0000 7f4cbac00700 1 -- start start 2024-04-16T11:32:18.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.739+0000 7f4cbac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb410f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:18.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.739+0000 7f4cbac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb410f610 con 0x7f4cb4071e20 2024-04-16T11:32:18.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.740+0000 7f4cb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb410f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:18.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.740+0000 7f4cb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb410f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33190/0 (socket says 172.21.15.44:33190) 2024-04-16T11:32:18.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.740+0000 7f4cb9800700 1 -- 172.21.15.44:0/849926350 learned_addr learned my addr 172.21.15.44:0/849926350 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:32:18.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.740+0000 7f4cb9800700 1 -- 172.21.15.44:0/849926350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8005040 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.740+0000 7f4cb9800700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb410f0d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800b900 tx=0x7f4ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.741+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ca800b5c0 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.741+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb410f810 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.741+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ca800be70 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.741+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ca801c590 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.741+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb4110380 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.742+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f4ca8004d30 con 0x7f4cb4071e20 2024-04-16T11:32:18.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.742+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb4104fb0 con 0x7f4cb4071e20 2024-04-16T11:32:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.743+0000 7f4cb1600700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f4c9c036880 0x7f4c9c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.743+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4ca80493a0 con 0x7f4cb4071e20 2024-04-16T11:32:18.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.743+0000 7f4cb8e00700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f4c9c036880 0x7f4c9c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:18.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.744+0000 7f4cb8e00700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f4c9c036880 0x7f4c9c038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009990 tx=0x7f4ca0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:18.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:18.748+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4ca801b350 con 0x7f4cb4071e20 2024-04-16T11:32:19.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.055+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cb405f690 con 0x7f4c9c036880 2024-04-16T11:32:19.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.058+0000 7f4cb1600700 1 -- 172.21.15.44:0/849926350 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f4cb405f690 con 0x7f4c9c036880 2024-04-16T11:32:19.056 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:32:19.056 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}] 2024-04-16T11:32:19.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.061+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f4c9c036880 msgr2=0x7f4c9c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:19.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.061+0000 7f4cbac00700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f4c9c036880 0x7f4c9c038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009990 tx=0x7f4ca0006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:19.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.061+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 msgr2=0x7f4cb410f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:19.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.061+0000 7f4cbac00700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb410f0d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800b900 tx=0x7f4ca8009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:19.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.062+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 shutdown_connections 2024-04-16T11:32:19.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.062+0000 7f4cbac00700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f4c9c036880 0x7f4c9c038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:19.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.062+0000 7f4cbac00700 1 --2- 172.21.15.44:0/849926350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4cb4071e20 0x7f4cb410f0d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:19.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.062+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 >> 172.21.15.44:0/849926350 conn(0x7f4cb40f8010 msgr2=0x7f4cb4105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:19.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.062+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 shutdown_connections 2024-04-16T11:32:19.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:19.062+0000 7f4cbac00700 1 -- 172.21.15.44:0/849926350 wait complete. 2024-04-16T11:32:20.127 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-16T11:32:20.128 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-16T11:32:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:19 smithi044 ceph-mon[28731]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:19 smithi044 ceph-mon[28731]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T11:32:21.568 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:32:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:21 smithi044 ceph-mon[28731]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:23.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:23 smithi044 ceph-mon[28731]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 -- 172.21.15.44:0/758783614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 msgr2=0x7fd3440ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 --2- 172.21.15.44:0/758783614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440ff660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd3340048c0 tx=0x7fd334009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 -- 172.21.15.44:0/758783614 shutdown_connections 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 --2- 172.21.15.44:0/758783614 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440ff660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 -- 172.21.15.44:0/758783614 >> 172.21.15.44:0/758783614 conn(0x7fd3440f70d0 msgr2=0x7fd3440f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 -- 172.21.15.44:0/758783614 shutdown_connections 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.778+0000 7fd349a00700 1 -- 172.21.15.44:0/758783614 wait complete. 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd349a00700 1 Processor -- start 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd349a00700 1 -- start start 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd349a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:25.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd349a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440fd4f0 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd342a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd342a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440fcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57056/0 (socket says 172.21.15.44:57056) 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.779+0000 7fd342a00700 1 -- 172.21.15.44:0/2861165110 learned_addr learned my addr 172.21.15.44:0/2861165110 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd342a00700 1 -- 172.21.15.44:0/2861165110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334005040 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd342a00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440fcfb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd33400b900 tx=0x7fd334009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33400b5c0 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd33400be70 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33401c590 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3440fb5c0 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.780+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3440fbab0 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.781+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fd334004d30 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.782+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34405f5f0 con 0x7fd344065d60 2024-04-16T11:32:25.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.782+0000 7fd33aa00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd3280368d0 0x7fd328038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.782+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd334049440 con 0x7fd344065d60 2024-04-16T11:32:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.782+0000 7fd33be00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd3280368d0 0x7fd328038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.783+0000 7fd33be00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd3280368d0 0x7fd328038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009990 tx=0x7fd32c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:24.788+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd33401b350 con 0x7fd344065d60 2024-04-16T11:32:25.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.030+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd344106440 con 0x7fd344065d60 2024-04-16T11:32:25.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.971+0000 7fd33aa00700 1 -- 172.21.15.44:0/2861165110 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd334021030 con 0x7fd344065d60 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd3280368d0 msgr2=0x7fd328038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd3280368d0 0x7fd328038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009990 tx=0x7fd32c006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 msgr2=0x7fd3440fcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440fcfb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd33400b900 tx=0x7fd334009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 shutdown_connections 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd3280368d0 0x7fd328038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 --2- 172.21.15.44:0/2861165110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd344065d60 0x7fd3440fcfb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 >> 172.21.15.44:0/2861165110 conn(0x7fd3440f70d0 msgr2=0x7fd344103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 shutdown_connections 2024-04-16T11:32:25.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:32:25.976+0000 7fd349a00700 1 -- 172.21.15.44:0/2861165110 wait complete. 2024-04-16T11:32:25.976 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-04-16T11:32:26.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:25 smithi044 ceph-mon[28731]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:26.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:25 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2861165110' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-16T11:32:26.729 INFO:tasks.cephadm:Adding mon.smithi044 on smithi044 2024-04-16T11:32:26.729 INFO:tasks.cephadm:Adding mon.smithi045 on smithi045 2024-04-16T11:32:26.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch apply mon '2;smithi044:172.21.15.44=smithi044;smithi045:172.21.15.45=smithi045' 2024-04-16T11:32:27.206 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:26 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2861165110' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-16T11:32:27.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:26 smithi044 ceph-mon[28731]: osdmap e4: 0 total, 0 up, 0 in 2024-04-16T11:32:27.323 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:27.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:27 smithi044 ceph-mon[28731]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:29.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.373+0000 7f77f7200700 1 -- 172.21.15.45:0/2004163074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 msgr2=0x7f77f80fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:29.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.373+0000 7f77f7200700 1 --2- 172.21.15.45:0/2004163074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f80fcca0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009c60 tx=0x7f77e8004d10 comp rx=0 tx=0).stop 2024-04-16T11:32:29.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.374+0000 7f77f7200700 1 -- 172.21.15.45:0/2004163074 shutdown_connections 2024-04-16T11:32:29.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.374+0000 7f77f7200700 1 --2- 172.21.15.45:0/2004163074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f80fcca0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:29.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.374+0000 7f77f7200700 1 -- 172.21.15.45:0/2004163074 >> 172.21.15.45:0/2004163074 conn(0x7f77f80f7ff0 msgr2=0x7f77f80fa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:29.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.374+0000 7f77f7200700 1 -- 172.21.15.45:0/2004163074 shutdown_connections 2024-04-16T11:32:29.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.374+0000 7f77f7200700 1 -- 172.21.15.45:0/2004163074 wait complete. 2024-04-16T11:32:29.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.375+0000 7f77f7200700 1 Processor -- start 2024-04-16T11:32:29.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.375+0000 7f77f7200700 1 -- start start 2024-04-16T11:32:29.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.376+0000 7f77f7200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f810f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:29.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.376+0000 7f77f7200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f810f640 con 0x7f77f80fc890 2024-04-16T11:32:29.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.376+0000 7f77f5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f810f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:29.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.376+0000 7f77f5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f810f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:57448/0 (socket says 172.21.15.45:57448) 2024-04-16T11:32:29.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.376+0000 7f77f5e00700 1 -- 172.21.15.45:0/4271237883 learned_addr learned my addr 172.21.15.45:0/4271237883 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:29.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.377+0000 7f77f5e00700 1 -- 172.21.15.45:0/4271237883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e8009940 con 0x7f77f80fc890 2024-04-16T11:32:29.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.377+0000 7f77f5e00700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f810f100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f77e8004e60 tx=0x7f77e8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:29.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.378+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e8012410 con 0x7f77f80fc890 2024-04-16T11:32:29.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.378+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f810f840 con 0x7f77f80fc890 2024-04-16T11:32:29.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.378+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77e8012a40 con 0x7f77f80fc890 2024-04-16T11:32:29.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.378+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e801ba40 con 0x7f77f80fc890 2024-04-16T11:32:29.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.378+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f81103b0 con 0x7f77f80fc890 2024-04-16T11:32:29.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.379+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f8102da0 con 0x7f77f80fc890 2024-04-16T11:32:29.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.380+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f77e8019040 con 0x7f77f80fc890 2024-04-16T11:32:29.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.380+0000 7f77ee000700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f77dc036920 0x7f77dc038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:29.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.381+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f77e8049d10 con 0x7f77f80fc890 2024-04-16T11:32:29.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.381+0000 7f77f5400700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f77dc036920 0x7f77dc038dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:29.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.382+0000 7f77f5400700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f77dc036920 0x7f77dc038dd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009990 tx=0x7f77e4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:29.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.389+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f77e801b3f0 con 0x7f77f80fc890 2024-04-16T11:32:29.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.616+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 --> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi044:172.21.15.44=smithi044;smithi045:172.21.15.45=smithi045", "target": ["mon-mgr", ""]}) v1 -- 0x7f77f8071720 con 0x7f77dc036920 2024-04-16T11:32:29.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.624+0000 7f77ee000700 1 -- 172.21.15.45:0/4271237883 <== mgr.14162 v2:172.21.15.44:6800/3690066543 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f77f8071720 con 0x7f77dc036920 2024-04-16T11:32:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:Scheduled mon update... 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f77dc036920 msgr2=0x7f77dc038dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f77dc036920 0x7f77dc038dd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009990 tx=0x7f77e4006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 msgr2=0x7f77f810f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f810f100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f77e8004e60 tx=0x7f77e8004d10 comp rx=0 tx=0).stop 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 shutdown_connections 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f77dc036920 0x7f77dc038dd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 --2- 172.21.15.45:0/4271237883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77f80fc890 0x7f77f810f100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.626+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 >> 172.21.15.45:0/4271237883 conn(0x7f77f80f7ff0 msgr2=0x7f77f80fa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.627+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 shutdown_connections 2024-04-16T11:32:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:29.627+0000 7f77f7200700 1 -- 172.21.15.45:0/4271237883 wait complete. 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: Deploying daemon grafana.smithi044 on smithi044 2024-04-16T11:32:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:29 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:30.347 DEBUG:teuthology.orchestra.run.smithi045:mon.smithi045> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi045.service 2024-04-16T11:32:30.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:30.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:30.854 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:30.959 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:30 smithi044 ceph-mon[28731]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi044:172.21.15.44=smithi044;smithi045:172.21.15.45=smithi045", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:32:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:30 smithi044 ceph-mon[28731]: Saving service mon spec with placement smithi044:172.21.15.44=smithi044;smithi045:172.21.15.45=smithi045;count:2 2024-04-16T11:32:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:30 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:31 smithi044 ceph-mon[28731]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:33.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 -- 172.21.15.45:0/3932894521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 msgr2=0x7fbbf00703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:33.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 --2- 172.21.15.45:0/3932894521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf00703b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40048c0 tx=0x7fbbe4009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 -- 172.21.15.45:0/3932894521 shutdown_connections 2024-04-16T11:32:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 --2- 172.21.15.45:0/3932894521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf00703b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 -- 172.21.15.45:0/3932894521 >> 172.21.15.45:0/3932894521 conn(0x7fbbf00f81d0 msgr2=0x7fbbf00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 -- 172.21.15.45:0/3932894521 shutdown_connections 2024-04-16T11:32:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.169+0000 7fbbf8000700 1 -- 172.21.15.45:0/3932894521 wait complete. 2024-04-16T11:32:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.170+0000 7fbbf8000700 1 Processor -- start 2024-04-16T11:32:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.170+0000 7fbbf8000700 1 -- start start 2024-04-16T11:32:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.170+0000 7fbbf8000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf01136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.171+0000 7fbbf8000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf0113c10 con 0x7fbbf0071ea0 2024-04-16T11:32:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.171+0000 7fbbf5600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf01136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.171+0000 7fbbf5600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf01136d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:44356/0 (socket says 172.21.15.45:44356) 2024-04-16T11:32:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.171+0000 7fbbf5600700 1 -- 172.21.15.45:0/3552889932 learned_addr learned my addr 172.21.15.45:0/3552889932 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:33.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.172+0000 7fbbf5600700 1 -- 172.21.15.45:0/3552889932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe4005040 con 0x7fbbf0071ea0 2024-04-16T11:32:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.173+0000 7fbbf5600700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf01136d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400b900 tx=0x7fbbe4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.174+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbe400b5c0 con 0x7fbbf0071ea0 2024-04-16T11:32:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.174+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf0113e10 con 0x7fbbf0071ea0 2024-04-16T11:32:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.174+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe400be70 con 0x7fbbf0071ea0 2024-04-16T11:32:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.174+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbe401c590 con 0x7fbbf0071ea0 2024-04-16T11:32:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.174+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf0114980 con 0x7fbbf0071ea0 2024-04-16T11:32:33.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.175+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf01095b0 con 0x7fbbf0071ea0 2024-04-16T11:32:33.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.176+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fbbe4004d30 con 0x7fbbf0071ea0 2024-04-16T11:32:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.176+0000 7fbbed600700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fbbd80368d0 0x7fbbd8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:33.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.176+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbbe4049410 con 0x7fbbf0071ea0 2024-04-16T11:32:33.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.177+0000 7fbbf4c00700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fbbd80368d0 0x7fbbd8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:33.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.179+0000 7fbbf4c00700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fbbd80368d0 0x7fbbd8038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009990 tx=0x7fbbdc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:33.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.198+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbbe4024d30 con 0x7fbbf0071ea0 2024-04-16T11:32:33.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.506+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbf0115450 con 0x7fbbf0071ea0 2024-04-16T11:32:33.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.507+0000 7fbbed600700 1 -- 172.21.15.45:0/3552889932 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbbe402b070 con 0x7fbbf0071ea0 2024-04-16T11:32:33.506 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:32:33.507 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fbbd80368d0 msgr2=0x7fbbd8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fbbd80368d0 0x7fbbd8038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009990 tx=0x7fbbdc006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 msgr2=0x7fbbf01136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf01136d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400b900 tx=0x7fbbe4009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 shutdown_connections 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fbbd80368d0 0x7fbbd8038d80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:33.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 --2- 172.21.15.45:0/3552889932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbf0071ea0 0x7fbbf01136d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 >> 172.21.15.45:0/3552889932 conn(0x7fbbf00f81d0 msgr2=0x7fbbf010a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 shutdown_connections 2024-04-16T11:32:33.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:33.510+0000 7fbbf8000700 1 -- 172.21.15.45:0/3552889932 wait complete. 2024-04-16T11:32:33.511 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:32:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:33 smithi044 ceph-mon[28731]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:33 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/3552889932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:32:35.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:35.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:35.805 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:36.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:35 smithi044 ceph-mon[28731]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:37.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.773+0000 7fc1aa800700 1 -- 172.21.15.45:0/1787806261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 msgr2=0x7fc1a4103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:37.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.773+0000 7fc1aa800700 1 --2- 172.21.15.45:0/1787806261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4103570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc1980048c0 tx=0x7fc198009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:37.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.774+0000 7fc1aa800700 1 -- 172.21.15.45:0/1787806261 shutdown_connections 2024-04-16T11:32:37.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.774+0000 7fc1aa800700 1 --2- 172.21.15.45:0/1787806261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4103570 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.774+0000 7fc1aa800700 1 -- 172.21.15.45:0/1787806261 >> 172.21.15.45:0/1787806261 conn(0x7fc1a4072080 msgr2=0x7fc1a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.774+0000 7fc1aa800700 1 -- 172.21.15.45:0/1787806261 shutdown_connections 2024-04-16T11:32:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.774+0000 7fc1aa800700 1 -- 172.21.15.45:0/1787806261 wait complete. 2024-04-16T11:32:37.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.775+0000 7fc1aa800700 1 Processor -- start 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.775+0000 7fc1aa800700 1 -- start start 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.775+0000 7fc1aa800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4113490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.775+0000 7fc1aa800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a41139d0 con 0x7fc1a4103160 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.776+0000 7fc1a9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.776+0000 7fc1a9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4113490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:44378/0 (socket says 172.21.15.45:44378) 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.776+0000 7fc1a9400700 1 -- 172.21.15.45:0/760192085 learned_addr learned my addr 172.21.15.45:0/760192085 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:37.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.777+0000 7fc1a9400700 1 -- 172.21.15.45:0/760192085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198005040 con 0x7fc1a4103160 2024-04-16T11:32:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.777+0000 7fc1a9400700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4113490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc198009b80 tx=0x7fc198009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.777+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc19800b5c0 con 0x7fc1a4103160 2024-04-16T11:32:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.777+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a4113bd0 con 0x7fc1a4103160 2024-04-16T11:32:37.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.777+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc19800be70 con 0x7fc1a4103160 2024-04-16T11:32:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.777+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc19801c570 con 0x7fc1a4103160 2024-04-16T11:32:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.778+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a4114740 con 0x7fc1a4103160 2024-04-16T11:32:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.779+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a4109390 con 0x7fc1a4103160 2024-04-16T11:32:37.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.779+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fc198004d90 con 0x7fc1a4103160 2024-04-16T11:32:37.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.780+0000 7fc1a1600700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc18c0368a0 0x7fc18c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:37.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.780+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc198049a70 con 0x7fc1a4103160 2024-04-16T11:32:37.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.780+0000 7fc1a8a00700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc18c0368a0 0x7fc18c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:37.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.782+0000 7fc1a8a00700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc18c0368a0 0x7fc18c038d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc190009990 tx=0x7fc190006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:37.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:37.797+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc198011960 con 0x7fc1a4103160 2024-04-16T11:32:38.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.097+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a4115210 con 0x7fc1a4103160 2024-04-16T11:32:38.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.098+0000 7fc1a1600700 1 -- 172.21.15.45:0/760192085 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc19801b390 con 0x7fc1a4103160 2024-04-16T11:32:38.097 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:32:38.097 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc18c0368a0 msgr2=0x7fc18c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc18c0368a0 0x7fc18c038d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc190009990 tx=0x7fc190006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 msgr2=0x7fc1a4113490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4113490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc198009b80 tx=0x7fc198009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 shutdown_connections 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc18c0368a0 0x7fc18c038d50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:38.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 --2- 172.21.15.45:0/760192085 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1a4103160 0x7fc1a4113490 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 >> 172.21.15.45:0/760192085 conn(0x7fc1a4072080 msgr2=0x7fc1a4109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 shutdown_connections 2024-04-16T11:32:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:38.101+0000 7fc1aa800700 1 -- 172.21.15.45:0/760192085 wait complete. 2024-04-16T11:32:38.101 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:32:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:37 smithi044 ceph-mon[28731]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:39.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:38 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/760192085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:32:39.754 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:39.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:40.332 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:40.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:39 smithi044 ceph-mon[28731]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 -- 172.21.15.45:0/3553849730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 msgr2=0x7ff2f00703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 --2- 172.21.15.45:0/3553849730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f00703b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40048c0 tx=0x7ff2e4009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 -- 172.21.15.45:0/3553849730 shutdown_connections 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 --2- 172.21.15.45:0/3553849730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f00703b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 -- 172.21.15.45:0/3553849730 >> 172.21.15.45:0/3553849730 conn(0x7ff2f00f81c0 msgr2=0x7ff2f00fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 -- 172.21.15.45:0/3553849730 shutdown_connections 2024-04-16T11:32:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.167+0000 7ff2f7e00700 1 -- 172.21.15.45:0/3553849730 wait complete. 2024-04-16T11:32:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.169+0000 7ff2f7e00700 1 Processor -- start 2024-04-16T11:32:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.169+0000 7ff2f7e00700 1 -- start start 2024-04-16T11:32:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.169+0000 7ff2f7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f010f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:42.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.169+0000 7ff2f7e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f010f870 con 0x7ff2f0071ea0 2024-04-16T11:32:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.170+0000 7ff2f5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f010f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.170+0000 7ff2f5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f010f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:34476/0 (socket says 172.21.15.45:34476) 2024-04-16T11:32:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.170+0000 7ff2f5400700 1 -- 172.21.15.45:0/1531861556 learned_addr learned my addr 172.21.15.45:0/1531861556 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.170+0000 7ff2f5400700 1 -- 172.21.15.45:0/1531861556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4005040 con 0x7ff2f0071ea0 2024-04-16T11:32:42.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.170+0000 7ff2f5400700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f010f330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400b900 tx=0x7ff2e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.171+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e400b5c0 con 0x7ff2f0071ea0 2024-04-16T11:32:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.171+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f010fa70 con 0x7ff2f0071ea0 2024-04-16T11:32:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.171+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2e400be70 con 0x7ff2f0071ea0 2024-04-16T11:32:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.171+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e401c590 con 0x7ff2f0071ea0 2024-04-16T11:32:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.171+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f01105e0 con 0x7ff2f0071ea0 2024-04-16T11:32:42.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.172+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f0105180 con 0x7ff2f0071ea0 2024-04-16T11:32:42.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.173+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7ff2e4004d30 con 0x7ff2f0071ea0 2024-04-16T11:32:42.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.173+0000 7ff2ed600700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7ff2d80368d0 0x7ff2d8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:42.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.173+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff2e4049410 con 0x7ff2f0071ea0 2024-04-16T11:32:42.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.174+0000 7ff2f4a00700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7ff2d80368d0 0x7ff2d8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:42.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.174+0000 7ff2f4a00700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7ff2d80368d0 0x7ff2d8038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009990 tx=0x7ff2dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:42.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.194+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff2e401b350 con 0x7ff2f0071ea0 2024-04-16T11:32:42.283 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:41 smithi044 ceph-mon[28731]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:42.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.493+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f005f5f0 con 0x7ff2f0071ea0 2024-04-16T11:32:42.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.494+0000 7ff2ed600700 1 -- 172.21.15.45:0/1531861556 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff2e4021030 con 0x7ff2f0071ea0 2024-04-16T11:32:42.492 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:32:42.492 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:32:42.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7ff2d80368d0 msgr2=0x7ff2d8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:42.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7ff2d80368d0 0x7ff2d8038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009990 tx=0x7ff2dc006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:42.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 msgr2=0x7ff2f010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f010f330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400b900 tx=0x7ff2e4009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 shutdown_connections 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7ff2d80368d0 0x7ff2d8038d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 --2- 172.21.15.45:0/1531861556 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff2f0071ea0 0x7ff2f010f330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.496+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 >> 172.21.15.45:0/1531861556 conn(0x7ff2f00f81c0 msgr2=0x7ff2f0105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.497+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 shutdown_connections 2024-04-16T11:32:42.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:42.497+0000 7ff2f7e00700 1 -- 172.21.15.45:0/1531861556 wait complete. 2024-04-16T11:32:42.496 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:32:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:42 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/1531861556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:32:44.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:44.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:43 smithi044 ceph-mon[28731]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:44.673 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:44.789 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:46.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:45 smithi044 ceph-mon[28731]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 -- 172.21.15.45:0/3677216484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 msgr2=0x7f5814103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 --2- 172.21.15.45:0/3677216484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814103570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f58040048c0 tx=0x7f5804009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 -- 172.21.15.45:0/3677216484 shutdown_connections 2024-04-16T11:32:46.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 --2- 172.21.15.45:0/3677216484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814103570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 -- 172.21.15.45:0/3677216484 >> 172.21.15.45:0/3677216484 conn(0x7f5814072080 msgr2=0x7f5814074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 -- 172.21.15.45:0/3677216484 shutdown_connections 2024-04-16T11:32:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.625+0000 7f5813200700 1 -- 172.21.15.45:0/3677216484 wait complete. 2024-04-16T11:32:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.626+0000 7f5813200700 1 Processor -- start 2024-04-16T11:32:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.626+0000 7f5813200700 1 -- start start 2024-04-16T11:32:46.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.627+0000 7f5813200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814113490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.627+0000 7f5813200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58141139d0 con 0x7f5814103160 2024-04-16T11:32:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.627+0000 7f5811e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.627+0000 7f5811e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814113490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:34488/0 (socket says 172.21.15.45:34488) 2024-04-16T11:32:46.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.627+0000 7f5811e00700 1 -- 172.21.15.45:0/1899384848 learned_addr learned my addr 172.21.15.45:0/1899384848 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.628+0000 7f5811e00700 1 -- 172.21.15.45:0/1899384848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804005040 con 0x7f5814103160 2024-04-16T11:32:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.628+0000 7f5811e00700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814113490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5804009b80 tx=0x7f5804009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:46.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.629+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580400b5c0 con 0x7f5814103160 2024-04-16T11:32:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.629+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5814113bd0 con 0x7f5814103160 2024-04-16T11:32:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.629+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f580400be70 con 0x7f5814103160 2024-04-16T11:32:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.629+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580401c570 con 0x7f5814103160 2024-04-16T11:32:46.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.629+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5814114740 con 0x7f5814103160 2024-04-16T11:32:46.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.630+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5814109390 con 0x7f5814103160 2024-04-16T11:32:46.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.630+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f5804004d90 con 0x7f5814103160 2024-04-16T11:32:46.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.631+0000 7f580a000700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f57f80368a0 0x7f57f8038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:46.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.631+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5804049b60 con 0x7f5814103160 2024-04-16T11:32:46.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.631+0000 7f5811400700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f57f80368a0 0x7f57f8038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:46.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.633+0000 7f5811400700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f57f80368a0 0x7f57f8038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5800009990 tx=0x7f5800006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:46.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.646+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5804011960 con 0x7f5814103160 2024-04-16T11:32:46.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.953+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5814115210 con 0x7f5814103160 2024-04-16T11:32:46.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.954+0000 7f580a000700 1 -- 172.21.15.45:0/1899384848 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f580401b390 con 0x7f5814103160 2024-04-16T11:32:46.952 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:32:46.952 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:32:46.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f57f80368a0 msgr2=0x7f57f8038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f57f80368a0 0x7f57f8038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5800009990 tx=0x7f5800006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 msgr2=0x7f5814113490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814113490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5804009b80 tx=0x7f5804009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 shutdown_connections 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f57f80368a0 0x7f57f8038d50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 --2- 172.21.15.45:0/1899384848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5814103160 0x7f5814113490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.957+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 >> 172.21.15.45:0/1899384848 conn(0x7f5814072080 msgr2=0x7f5814109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.958+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 shutdown_connections 2024-04-16T11:32:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:46.958+0000 7f5813200700 1 -- 172.21.15.45:0/1899384848 wait complete. 2024-04-16T11:32:46.957 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:32:47.128 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:47 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/1899384848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:32:48.139 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:48 smithi044 ceph-mon[28731]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:48.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:48.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:49.073 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:49.184 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:50.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:50 smithi044 ceph-mon[28731]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.101+0000 7f9f3ae00700 1 -- 172.21.15.45:0/20850276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 msgr2=0x7f9f340feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.101+0000 7f9f3ae00700 1 --2- 172.21.15.45:0/20850276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f340feef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f280048c0 tx=0x7f9f28009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.102+0000 7f9f3ae00700 1 -- 172.21.15.45:0/20850276 shutdown_connections 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.102+0000 7f9f3ae00700 1 --2- 172.21.15.45:0/20850276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f340feef0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.102+0000 7f9f3ae00700 1 -- 172.21.15.45:0/20850276 >> 172.21.15.45:0/20850276 conn(0x7f9f340fa5d0 msgr2=0x7f9f340fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.102+0000 7f9f3ae00700 1 -- 172.21.15.45:0/20850276 shutdown_connections 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.102+0000 7f9f3ae00700 1 -- 172.21.15.45:0/20850276 wait complete. 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.103+0000 7f9f3ae00700 1 Processor -- start 2024-04-16T11:32:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.103+0000 7f9f3ae00700 1 -- start start 2024-04-16T11:32:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.103+0000 7f9f3ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f3410f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.104+0000 7f9f3ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3410f630 con 0x7f9f340feae0 2024-04-16T11:32:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.104+0000 7f9f39a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f3410f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.104+0000 7f9f39a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f3410f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:57936/0 (socket says 172.21.15.45:57936) 2024-04-16T11:32:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.104+0000 7f9f39a00700 1 -- 172.21.15.45:0/2058968990 learned_addr learned my addr 172.21.15.45:0/2058968990 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.105+0000 7f9f39a00700 1 -- 172.21.15.45:0/2058968990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f28005040 con 0x7f9f340feae0 2024-04-16T11:32:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.105+0000 7f9f39a00700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f3410f0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009680 tx=0x7f9f28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.106+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f2800b5c0 con 0x7f9f340feae0 2024-04-16T11:32:51.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.106+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3410f830 con 0x7f9f340feae0 2024-04-16T11:32:51.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.106+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f341103a0 con 0x7f9f340feae0 2024-04-16T11:32:51.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.106+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f2800be70 con 0x7f9f340feae0 2024-04-16T11:32:51.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.106+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f280135c0 con 0x7f9f340feae0 2024-04-16T11:32:51.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.107+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f34104ff0 con 0x7f9f340feae0 2024-04-16T11:32:51.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.107+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f9f28004d30 con 0x7f9f340feae0 2024-04-16T11:32:51.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.108+0000 7f9f31600700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f9f1c036830 0x7f9f1c038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.108+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9f28049440 con 0x7f9f340feae0 2024-04-16T11:32:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.108+0000 7f9f39000700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f9f1c036830 0x7f9f1c038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.109+0000 7f9f39000700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f9f1c036830 0x7f9f1c038ce0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009990 tx=0x7f9f20006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.117+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9f28024be0 con 0x7f9f340feae0 2024-04-16T11:32:51.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.419+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f3405f5f0 con 0x7f9f340feae0 2024-04-16T11:32:51.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.421+0000 7f9f31600700 1 -- 172.21.15.45:0/2058968990 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9f28021020 con 0x7f9f340feae0 2024-04-16T11:32:51.419 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:32:51.419 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.423+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f9f1c036830 msgr2=0x7f9f1c038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.423+0000 7f9f3ae00700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f9f1c036830 0x7f9f1c038ce0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009990 tx=0x7f9f20006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.423+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 msgr2=0x7f9f3410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.423+0000 7f9f3ae00700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f3410f0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009680 tx=0x7f9f28009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.424+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 shutdown_connections 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.424+0000 7f9f3ae00700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f9f1c036830 0x7f9f1c038ce0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.424+0000 7f9f3ae00700 1 --2- 172.21.15.45:0/2058968990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f340feae0 0x7f9f3410f0f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.424+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 >> 172.21.15.45:0/2058968990 conn(0x7f9f340fa5d0 msgr2=0x7f9f340fb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.424+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 shutdown_connections 2024-04-16T11:32:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:51.424+0000 7f9f3ae00700 1 -- 172.21.15.45:0/2058968990 wait complete. 2024-04-16T11:32:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: Deploying daemon prometheus.smithi044 on smithi044 2024-04-16T11:32:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:52 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2058968990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:32:53.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:53.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:53.511 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:53.613 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:54.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:54 smithi044 ceph-mon[28731]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.555+0000 7feb09e00700 1 -- 172.21.15.45:0/3733091934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 msgr2=0x7feb04101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.555+0000 7feb09e00700 1 --2- 172.21.15.45:0/3733091934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb04101ac0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feaf40048c0 tx=0x7feaf4009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.556+0000 7feb09e00700 1 -- 172.21.15.45:0/3733091934 shutdown_connections 2024-04-16T11:32:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.556+0000 7feb09e00700 1 --2- 172.21.15.45:0/3733091934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb04101ac0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.556+0000 7feb09e00700 1 -- 172.21.15.45:0/3733091934 >> 172.21.15.45:0/3733091934 conn(0x7feb040f70d0 msgr2=0x7feb040f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:55.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.556+0000 7feb09e00700 1 -- 172.21.15.45:0/3733091934 shutdown_connections 2024-04-16T11:32:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.556+0000 7feb09e00700 1 -- 172.21.15.45:0/3733091934 wait complete. 2024-04-16T11:32:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.557+0000 7feb09e00700 1 Processor -- start 2024-04-16T11:32:55.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.558+0000 7feb09e00700 1 -- start start 2024-04-16T11:32:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.558+0000 7feb09e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb0410f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:55.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.558+0000 7feb09e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0410f810 con 0x7feb04065d00 2024-04-16T11:32:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.559+0000 7feb02a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb0410f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.559+0000 7feb02a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb0410f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:57946/0 (socket says 172.21.15.45:57946) 2024-04-16T11:32:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.559+0000 7feb02a00700 1 -- 172.21.15.45:0/134301642 learned_addr learned my addr 172.21.15.45:0/134301642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:32:55.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.560+0000 7feb02a00700 1 -- 172.21.15.45:0/134301642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4005040 con 0x7feb04065d00 2024-04-16T11:32:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.560+0000 7feb02a00700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb0410f2d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009680 tx=0x7feaf4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.561+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaf400bed0 con 0x7feb04065d00 2024-04-16T11:32:55.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.561+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0410fa10 con 0x7feb04065d00 2024-04-16T11:32:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.561+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feaf400b5c0 con 0x7feb04065d00 2024-04-16T11:32:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.561+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaf4025560 con 0x7feb04065d00 2024-04-16T11:32:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.561+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb04110580 con 0x7feb04065d00 2024-04-16T11:32:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.562+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb04105180 con 0x7feb04065d00 2024-04-16T11:32:55.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.562+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7feaf4013a80 con 0x7feb04065d00 2024-04-16T11:32:55.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.563+0000 7feafaa00700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7feae8036880 0x7feae8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:32:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.563+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7feaf4049f30 con 0x7feb04065d00 2024-04-16T11:32:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.564+0000 7feb02000700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7feae8036880 0x7feae8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:32:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.565+0000 7feb02000700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7feae8036880 0x7feae8038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009990 tx=0x7feaf0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:32:55.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.586+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7feaf40238f0 con 0x7feb04065d00 2024-04-16T11:32:55.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.891+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb0404bed0 con 0x7feb04065d00 2024-04-16T11:32:55.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.893+0000 7feafaa00700 1 -- 172.21.15.45:0/134301642 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7feaf4023300 con 0x7feb04065d00 2024-04-16T11:32:55.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:32:55.891 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:32:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7feae8036880 msgr2=0x7feae8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7feae8036880 0x7feae8038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009990 tx=0x7feaf0006e40 comp rx=0 tx=0).stop 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 msgr2=0x7feb0410f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb0410f2d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009680 tx=0x7feaf4009e30 comp rx=0 tx=0).stop 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 shutdown_connections 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7feae8036880 0x7feae8038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 --2- 172.21.15.45:0/134301642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb04065d00 0x7feb0410f2d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.895+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 >> 172.21.15.45:0/134301642 conn(0x7feb040f70d0 msgr2=0x7feb04105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.896+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 shutdown_connections 2024-04-16T11:32:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:32:55.896+0000 7feb09e00700 1 -- 172.21.15.45:0/134301642 wait complete. 2024-04-16T11:32:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:32:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:55 smithi044 ceph-mon[28731]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:32:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:55 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:32:56.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:55 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/134301642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:32:57.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:32:57.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:32:58.019 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:58.120 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:32:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:57 smithi044 ceph-mon[28731]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:00.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 -- 172.21.15.45:0/1412519861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 msgr2=0x7f76b80fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:00.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 --2- 172.21.15.45:0/1412519861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b80fcca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0048c0 tx=0x7f76ac009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 -- 172.21.15.45:0/1412519861 shutdown_connections 2024-04-16T11:33:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 --2- 172.21.15.45:0/1412519861 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b80fcca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 -- 172.21.15.45:0/1412519861 >> 172.21.15.45:0/1412519861 conn(0x7f76b80f7ff0 msgr2=0x7f76b80fa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 -- 172.21.15.45:0/1412519861 shutdown_connections 2024-04-16T11:33:00.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.035+0000 7f76bde00700 1 -- 172.21.15.45:0/1412519861 wait complete. 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.036+0000 7f76bde00700 1 Processor -- start 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.036+0000 7f76bde00700 1 -- start start 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.037+0000 7f76bde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b810f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.037+0000 7f76bde00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b810f5f0 con 0x7f76b80fc890 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.037+0000 7f76bca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b810f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.037+0000 7f76bca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b810f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:33106/0 (socket says 172.21.15.45:33106) 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.037+0000 7f76bca00700 1 -- 172.21.15.45:0/4127857057 learned_addr learned my addr 172.21.15.45:0/4127857057 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:00.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76bca00700 1 -- 172.21.15.45:0/4127857057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac005040 con 0x7f76b80fc890 2024-04-16T11:33:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76bca00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b810f0b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009680 tx=0x7f76ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac00b5c0 con 0x7f76b80fc890 2024-04-16T11:33:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b810f7f0 con 0x7f76b80fc890 2024-04-16T11:33:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76ac00be70 con 0x7f76b80fc890 2024-04-16T11:33:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac0135a0 con 0x7f76b80fc890 2024-04-16T11:33:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.038+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b8110360 con 0x7f76b80fc890 2024-04-16T11:33:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.039+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b8102da0 con 0x7f76b80fc890 2024-04-16T11:33:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.040+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f76ac004d30 con 0x7f76b80fc890 2024-04-16T11:33:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.040+0000 7f76b4c00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f76a0036880 0x7f76a0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.040+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f76ac0493a0 con 0x7f76b80fc890 2024-04-16T11:33:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.041+0000 7f76b7e00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f76a0036880 0x7f76a0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.042+0000 7f76b7e00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f76a0036880 0x7f76a0038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009990 tx=0x7f76a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:00.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.056+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f76ac011960 con 0x7f76b80fc890 2024-04-16T11:33:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:32:59 smithi044 ceph-mon[28731]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:00.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.351+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76b8110660 con 0x7f76b80fc890 2024-04-16T11:33:00.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.352+0000 7f76b4c00700 1 -- 172.21.15.45:0/4127857057 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f76ac021020 con 0x7f76b80fc890 2024-04-16T11:33:00.350 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:00.350 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.354+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f76a0036880 msgr2=0x7f76a0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.354+0000 7f76bde00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f76a0036880 0x7f76a0038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009990 tx=0x7f76a4006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.354+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 msgr2=0x7f76b810f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.354+0000 7f76bde00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b810f0b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009680 tx=0x7f76ac009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.355+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 shutdown_connections 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.355+0000 7f76bde00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f76a0036880 0x7f76a0038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.355+0000 7f76bde00700 1 --2- 172.21.15.45:0/4127857057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76b80fc890 0x7f76b810f0b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.355+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 >> 172.21.15.45:0/4127857057 conn(0x7f76b80f7ff0 msgr2=0x7f76b80fa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.355+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 shutdown_connections 2024-04-16T11:33:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:00.355+0000 7f76bde00700 1 -- 172.21.15.45:0/4127857057 wait complete. 2024-04-16T11:33:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:01.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:00 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/4127857057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:02.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:02.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:02.441 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:01 smithi044 ceph-mon[28731]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:02.680 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:04.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:03 smithi044 ceph-mon[28731]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:04.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.683+0000 7f2671400700 1 -- 172.21.15.45:0/3454761848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 msgr2=0x7f266c103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.683+0000 7f2671400700 1 --2- 172.21.15.45:0/3454761848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c103570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f265c0048c0 tx=0x7f265c009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.683+0000 7f2671400700 1 -- 172.21.15.45:0/3454761848 shutdown_connections 2024-04-16T11:33:04.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.683+0000 7f2671400700 1 --2- 172.21.15.45:0/3454761848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c103570 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.683+0000 7f2671400700 1 -- 172.21.15.45:0/3454761848 >> 172.21.15.45:0/3454761848 conn(0x7f266c072080 msgr2=0x7f266c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.684+0000 7f2671400700 1 -- 172.21.15.45:0/3454761848 shutdown_connections 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.684+0000 7f2671400700 1 -- 172.21.15.45:0/3454761848 wait complete. 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.684+0000 7f2671400700 1 Processor -- start 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.684+0000 7f2671400700 1 -- start start 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.685+0000 7f2671400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c113490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.685+0000 7f2671400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1139d0 con 0x7f266c103160 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.685+0000 7f266be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.685+0000 7f266be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c113490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:33132/0 (socket says 172.21.15.45:33132) 2024-04-16T11:33:04.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.685+0000 7f266be00700 1 -- 172.21.15.45:0/2620586823 learned_addr learned my addr 172.21.15.45:0/2620586823 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.686+0000 7f266be00700 1 -- 172.21.15.45:0/2620586823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c005040 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.686+0000 7f266be00700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c113490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f265c009b80 tx=0x7f265c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.686+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f265c00b5c0 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.686+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c113bd0 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.687+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f265c00be70 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.687+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f265c01c570 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.687+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c114740 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.688+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c109390 con 0x7f266c103160 2024-04-16T11:33:04.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.688+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f265c004d90 con 0x7f266c103160 2024-04-16T11:33:04.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.688+0000 7f2663e00700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f2650036850 0x7f2650038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:04.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.689+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f265c049b60 con 0x7f266c103160 2024-04-16T11:33:04.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.689+0000 7f266b400700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f2650036850 0x7f2650038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:04.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.690+0000 7f266b400700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f2650036850 0x7f2650038d00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2658009990 tx=0x7f2658006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:04.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:04.701+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f265c011960 con 0x7f266c103160 2024-04-16T11:33:05.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.006+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f266c115210 con 0x7f266c103160 2024-04-16T11:33:05.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.007+0000 7f2663e00700 1 -- 172.21.15.45:0/2620586823 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f265c01b390 con 0x7f266c103160 2024-04-16T11:33:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:05.006 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f2650036850 msgr2=0x7f2650038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f2650036850 0x7f2650038d00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2658009990 tx=0x7f2658006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 msgr2=0x7f266c113490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c113490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f265c009b80 tx=0x7f265c009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 shutdown_connections 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7f2650036850 0x7f2650038d00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 --2- 172.21.15.45:0/2620586823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f266c103160 0x7f266c113490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 >> 172.21.15.45:0/2620586823 conn(0x7f266c072080 msgr2=0x7f266c109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:05.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 shutdown_connections 2024-04-16T11:33:05.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:05.010+0000 7f2671400700 1 -- 172.21.15.45:0/2620586823 wait complete. 2024-04-16T11:33:05.009 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:06.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:05 smithi044 ceph-mon[28731]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:05 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2620586823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:05 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:05 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:05 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:05 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-16T11:33:06.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:06.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:07.213 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:07.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:07 smithi044 ceph-mon[28731]: from='mgr.14162 172.21.15.44:0/3737064746' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-16T11:33:07.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:07 smithi044 ceph-mon[28731]: mgrmap e14: smithi044.sczobv(active, since 82s) 2024-04-16T11:33:09.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 -- 172.21.15.45:0/1041660700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 msgr2=0x7fd8500ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:09.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 --2- 172.21.15.45:0/1041660700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd8500ff080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8400048c0 tx=0x7fd840009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:09.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 -- 172.21.15.45:0/1041660700 shutdown_connections 2024-04-16T11:33:09.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 --2- 172.21.15.45:0/1041660700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd8500ff080 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:09.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 -- 172.21.15.45:0/1041660700 >> 172.21.15.45:0/1041660700 conn(0x7fd8500fa5d0 msgr2=0x7fd8500fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:09.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 -- 172.21.15.45:0/1041660700 shutdown_connections 2024-04-16T11:33:09.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.418+0000 7fd857200700 1 -- 172.21.15.45:0/1041660700 wait complete. 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.419+0000 7fd857200700 1 Processor -- start 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.419+0000 7fd857200700 1 -- start start 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.420+0000 7fd857200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd85010f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.420+0000 7fd857200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85010f830 con 0x7fd8500fec70 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.420+0000 7fd84fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd85010f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.420+0000 7fd84fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd85010f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:33142/0 (socket says 172.21.15.45:33142) 2024-04-16T11:33:09.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.420+0000 7fd84fe00700 1 -- 172.21.15.45:0/4138327996 learned_addr learned my addr 172.21.15.45:0/4138327996 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:09.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.421+0000 7fd84fe00700 1 -- 172.21.15.45:0/4138327996 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd840005040 con 0x7fd8500fec70 2024-04-16T11:33:09.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.421+0000 7fd84fe00700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd85010f2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8400096b0 tx=0x7fd840009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:09.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.421+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84000bed0 con 0x7fd8500fec70 2024-04-16T11:33:09.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.421+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85010fa30 con 0x7fd8500fec70 2024-04-16T11:33:09.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.422+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd84000b5c0 con 0x7fd8500fec70 2024-04-16T11:33:09.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.422+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84001c5b0 con 0x7fd8500fec70 2024-04-16T11:33:09.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.422+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd850110550 con 0x7fd8500fec70 2024-04-16T11:33:09.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.423+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd850105180 con 0x7fd8500fec70 2024-04-16T11:33:09.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.423+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7fd840013a80 con 0x7fd8500fec70 2024-04-16T11:33:09.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.423+0000 7fd847e00700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 0x7fd834038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:09.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.423+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd840049b60 con 0x7fd8500fec70 2024-04-16T11:33:09.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.424+0000 7fd84f400700 1 -- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 msgr2=0x7fd834038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3690066543 2024-04-16T11:33:09.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.424+0000 7fd84f400700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 0x7fd834038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:33:09.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.441+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd840011940 con 0x7fd8500fec70 2024-04-16T11:33:09.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.625+0000 7fd84f400700 1 -- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 msgr2=0x7fd834038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3690066543 2024-04-16T11:33:09.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.625+0000 7fd84f400700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 0x7fd834038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:33:09.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.748+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd850110f60 con 0x7fd8500fec70 2024-04-16T11:33:09.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.750+0000 7fd847e00700 1 -- 172.21.15.45:0/4138327996 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd840011370 con 0x7fd8500fec70 2024-04-16T11:33:09.748 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:09.748 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 msgr2=0x7fd834038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 0x7fd834038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 msgr2=0x7fd85010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd85010f2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8400096b0 tx=0x7fd840009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 shutdown_connections 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fd8340368d0 0x7fd834038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 --2- 172.21.15.45:0/4138327996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8500fec70 0x7fd85010f2f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 >> 172.21.15.45:0/4138327996 conn(0x7fd8500fa5d0 msgr2=0x7fd8500fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 shutdown_connections 2024-04-16T11:33:09.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:09.753+0000 7fd857200700 1 -- 172.21.15.45:0/4138327996 wait complete. 2024-04-16T11:33:09.753 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:10.098 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:09 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/4138327996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:11.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:11.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:11.931 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:12.036 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.097+0000 7fc49f400700 1 -- 172.21.15.45:0/2459608276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 msgr2=0x7fc4980ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.097+0000 7fc49f400700 1 --2- 172.21.15.45:0/2459608276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc4980ff080 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc48c0048c0 tx=0x7fc48c009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.097+0000 7fc49f400700 1 -- 172.21.15.45:0/2459608276 shutdown_connections 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.097+0000 7fc49f400700 1 --2- 172.21.15.45:0/2459608276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc4980ff080 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.098+0000 7fc49f400700 1 -- 172.21.15.45:0/2459608276 >> 172.21.15.45:0/2459608276 conn(0x7fc4980fa5d0 msgr2=0x7fc4980fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.098+0000 7fc49f400700 1 -- 172.21.15.45:0/2459608276 shutdown_connections 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.098+0000 7fc49f400700 1 -- 172.21.15.45:0/2459608276 wait complete. 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.099+0000 7fc49f400700 1 Processor -- start 2024-04-16T11:33:14.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.099+0000 7fc49f400700 1 -- start start 2024-04-16T11:33:14.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.099+0000 7fc49f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc49810f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:14.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.099+0000 7fc49f400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49810f830 con 0x7fc4980fec70 2024-04-16T11:33:14.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.100+0000 7fc49ca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc49810f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:14.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.100+0000 7fc49ca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc49810f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:36678/0 (socket says 172.21.15.45:36678) 2024-04-16T11:33:14.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.100+0000 7fc49ca00700 1 -- 172.21.15.45:0/1130586539 learned_addr learned my addr 172.21.15.45:0/1130586539 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:14.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.101+0000 7fc49ca00700 1 -- 172.21.15.45:0/1130586539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c005040 con 0x7fc4980fec70 2024-04-16T11:33:14.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.101+0000 7fc49ca00700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc49810f2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009680 tx=0x7fc48c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:14.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.102+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc48c00b5c0 con 0x7fc4980fec70 2024-04-16T11:33:14.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.102+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc48c00be70 con 0x7fc4980fec70 2024-04-16T11:33:14.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.102+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49810fa30 con 0x7fc4980fec70 2024-04-16T11:33:14.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.102+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc48c0135a0 con 0x7fc4980fec70 2024-04-16T11:33:14.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.102+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc498110550 con 0x7fc4980fec70 2024-04-16T11:33:14.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.103+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc498105180 con 0x7fc4980fec70 2024-04-16T11:33:14.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.104+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7fc48c01ca90 con 0x7fc4980fec70 2024-04-16T11:33:14.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.104+0000 7fc494c00700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 0x7fc480038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:14.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.104+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc48c049450 con 0x7fc4980fec70 2024-04-16T11:33:14.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.104+0000 7fc497e00700 1 -- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 msgr2=0x7fc480038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3690066543 2024-04-16T11:33:14.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.104+0000 7fc497e00700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 0x7fc480038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:33:14.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.111+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc48c011960 con 0x7fc4980fec70 2024-04-16T11:33:14.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.305+0000 7fc497e00700 1 -- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 msgr2=0x7fc480038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3690066543 2024-04-16T11:33:14.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.305+0000 7fc497e00700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 0x7fc480038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:33:14.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.408+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc498110f60 con 0x7fc4980fec70 2024-04-16T11:33:14.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.409+0000 7fc494c00700 1 -- 172.21.15.45:0/1130586539 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc48c021070 con 0x7fc4980fec70 2024-04-16T11:33:14.408 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:14.408 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:14.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 msgr2=0x7fc480038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:33:14.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 0x7fc480038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:14.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 msgr2=0x7fc49810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:14.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc49810f2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009680 tx=0x7fc48c009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 shutdown_connections 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:6800/3690066543,v1:172.21.15.44:6801/3690066543] conn(0x7fc480036920 0x7fc480038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 --2- 172.21.15.45:0/1130586539 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4980fec70 0x7fc49810f2f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 >> 172.21.15.45:0/1130586539 conn(0x7fc4980fa5d0 msgr2=0x7fc4980fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 shutdown_connections 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:14.413+0000 7fc49f400700 1 -- 172.21.15.45:0/1130586539 wait complete. 2024-04-16T11:33:14.412 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:14.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:14 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/1130586539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:16.038 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:16.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:16.079 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: Active manager daemon smithi044.sczobv restarted 2024-04-16T11:33:16.079 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: Activating manager daemon smithi044.sczobv 2024-04-16T11:33:16.079 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T11:33:16.079 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: mgrmap e15: smithi044.sczobv(active, starting, since 0.00347105s) 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:33:16.080 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:33:16.516 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:16.627 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:33:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:33:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:33:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:16 smithi044 ceph-mon[28731]: mgrmap e16: smithi044.sczobv(active, since 1.00805s) 2024-04-16T11:33:18.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:18 smithi044 ceph-mon[28731]: [16/Apr/2024:11:33:16] ENGINE Bus STARTING 2024-04-16T11:33:18.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:18 smithi044 ceph-mon[28731]: [16/Apr/2024:11:33:17] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:33:18.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:18 smithi044 ceph-mon[28731]: [16/Apr/2024:11:33:17] ENGINE Bus STARTED 2024-04-16T11:33:18.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:18 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:18.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:18 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 -- 172.21.15.45:0/1769441116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 msgr2=0x7ff9b4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 --2- 172.21.15.45:0/1769441116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b4070350 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40048c0 tx=0x7ff9a4009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 -- 172.21.15.45:0/1769441116 shutdown_connections 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 --2- 172.21.15.45:0/1769441116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b4070350 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 -- 172.21.15.45:0/1769441116 >> 172.21.15.45:0/1769441116 conn(0x7ff9b40f7ff0 msgr2=0x7ff9b40fa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 -- 172.21.15.45:0/1769441116 shutdown_connections 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.495+0000 7ff9b3e00700 1 -- 172.21.15.45:0/1769441116 wait complete. 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.496+0000 7ff9b3e00700 1 Processor -- start 2024-04-16T11:33:18.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.496+0000 7ff9b3e00700 1 -- start start 2024-04-16T11:33:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.496+0000 7ff9b3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b410f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.496+0000 7ff9b3e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b410f680 con 0x7ff9b4071e40 2024-04-16T11:33:18.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.497+0000 7ff9b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b410f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.497+0000 7ff9b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b410f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:36684/0 (socket says 172.21.15.45:36684) 2024-04-16T11:33:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.497+0000 7ff9b2a00700 1 -- 172.21.15.45:0/2633328925 learned_addr learned my addr 172.21.15.45:0/2633328925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:18.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.498+0000 7ff9b2a00700 1 -- 172.21.15.45:0/2633328925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4005040 con 0x7ff9b4071e40 2024-04-16T11:33:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.498+0000 7ff9b2a00700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b410f140 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009680 tx=0x7ff9a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:18.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.498+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a400bed0 con 0x7ff9b4071e40 2024-04-16T11:33:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.499+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b410f880 con 0x7ff9b4071e40 2024-04-16T11:33:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.499+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9a400b5c0 con 0x7ff9b4071e40 2024-04-16T11:33:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.499+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a401c5b0 con 0x7ff9b4071e40 2024-04-16T11:33:18.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.499+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b41103f0 con 0x7ff9b4071e40 2024-04-16T11:33:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.500+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b4104fb0 con 0x7ff9b4071e40 2024-04-16T11:33:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.500+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7ff9a4004d30 con 0x7ff9b4071e40 2024-04-16T11:33:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.501+0000 7ff9aaa00700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff9980369c0 0x7ff998038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.501+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff9a4049b40 con 0x7ff9b4071e40 2024-04-16T11:33:18.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.501+0000 7ff9b2000700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff9980369c0 0x7ff998038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:18.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.503+0000 7ff9b2000700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff9980369c0 0x7ff998038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009990 tx=0x7ff9a0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:18.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.516+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff9a4011960 con 0x7ff9b4071e40 2024-04-16T11:33:18.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.817+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9b405f640 con 0x7ff9b4071e40 2024-04-16T11:33:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.818+0000 7ff9aaa00700 1 -- 172.21.15.45:0/2633328925 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff9a4049e60 con 0x7ff9b4071e40 2024-04-16T11:33:18.816 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:18.816 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff9980369c0 msgr2=0x7ff998038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff9980369c0 0x7ff998038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009990 tx=0x7ff9a0006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 msgr2=0x7ff9b410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b410f140 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009680 tx=0x7ff9a4009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 shutdown_connections 2024-04-16T11:33:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff9980369c0 0x7ff998038e70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 --2- 172.21.15.45:0/2633328925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9b4071e40 0x7ff9b410f140 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 >> 172.21.15.45:0/2633328925 conn(0x7ff9b40f7ff0 msgr2=0x7ff9b4105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 shutdown_connections 2024-04-16T11:33:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:18.821+0000 7ff9b3e00700 1 -- 172.21.15.45:0/2633328925 wait complete. 2024-04-16T11:33:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:19.313 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:19 smithi044 ceph-mon[28731]: mgrmap e17: smithi044.sczobv(active, since 2s) 2024-04-16T11:33:19.313 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:19 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2633328925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:20.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:20.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:20.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:21.422 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 -- 172.21.15.45:0/1865463477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 msgr2=0x7f2a60103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 --2- 172.21.15.45:0/1865463477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a60103490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2a540048c0 tx=0x7f2a54009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 -- 172.21.15.45:0/1865463477 shutdown_connections 2024-04-16T11:33:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 --2- 172.21.15.45:0/1865463477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a60103490 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 -- 172.21.15.45:0/1865463477 >> 172.21.15.45:0/1865463477 conn(0x7f2a60072300 msgr2=0x7f2a60074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 -- 172.21.15.45:0/1865463477 shutdown_connections 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.839+0000 7f2a68400700 1 -- 172.21.15.45:0/1865463477 wait complete. 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.841+0000 7f2a68400700 1 Processor -- start 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.841+0000 7f2a68400700 1 -- start start 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.841+0000 7f2a68400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a601136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.841+0000 7f2a68400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60113c00 con 0x7f2a60103080 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.842+0000 7f2a65a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a601136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.842+0000 7f2a65a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a601136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:32848/0 (socket says 172.21.15.45:32848) 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.842+0000 7f2a65a00700 1 -- 172.21.15.45:0/4161793787 learned_addr learned my addr 172.21.15.45:0/4161793787 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.843+0000 7f2a65a00700 1 -- 172.21.15.45:0/4161793787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54005040 con 0x7f2a60103080 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.843+0000 7f2a65a00700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a601136c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009680 tx=0x7f2a54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.843+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a5400b5c0 con 0x7f2a60103080 2024-04-16T11:33:23.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.843+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a60113e00 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.843+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a5400be70 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.843+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a540135a0 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.844+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a60114970 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.845+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a60109570 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.845+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f2a54004d30 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.846+0000 7f2a5d600700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2a480369c0 0x7f2a48038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.846+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2a54049660 con 0x7f2a60103080 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.846+0000 7f2a65000700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2a480369c0 0x7f2a48038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.847+0000 7f2a65000700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2a480369c0 0x7f2a48038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009990 tx=0x7f2a4c006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:23.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:23.857+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2a54011960 con 0x7f2a60103080 2024-04-16T11:33:24.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.171+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a60115380 con 0x7f2a60103080 2024-04-16T11:33:24.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.173+0000 7f2a5d600700 1 -- 172.21.15.45:0/4161793787 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2a54021070 con 0x7f2a60103080 2024-04-16T11:33:24.171 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:24.171 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2a480369c0 msgr2=0x7f2a48038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2a480369c0 0x7f2a48038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009990 tx=0x7f2a4c006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 msgr2=0x7f2a601136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a601136c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009680 tx=0x7f2a54009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 shutdown_connections 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2a480369c0 0x7f2a48038e70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 --2- 172.21.15.45:0/4161793787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a60103080 0x7f2a601136c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:24.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 >> 172.21.15.45:0/4161793787 conn(0x7f2a60072300 msgr2=0x7f2a60074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:24.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 shutdown_connections 2024-04-16T11:33:24.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:24.176+0000 7f2a68400700 1 -- 172.21.15.45:0/4161793787 wait complete. 2024-04-16T11:33:24.176 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:24 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/4161793787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:25.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:25.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:25.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:25.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:25.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:25.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:25.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:33:26.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:26.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:26.498 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:26.597 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T11:33:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:33:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:33:27.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:33:27.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:33:27.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:26 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:33:28.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:27 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:33:28.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:27 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:33:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 -- 172.21.15.45:0/2378886282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 msgr2=0x7f385c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 --2- 172.21.15.45:0/2378886282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c070730 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f38500048c0 tx=0x7f3850009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 -- 172.21.15.45:0/2378886282 shutdown_connections 2024-04-16T11:33:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 --2- 172.21.15.45:0/2378886282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c070730 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 -- 172.21.15.45:0/2378886282 >> 172.21.15.45:0/2378886282 conn(0x7f385c0f8600 msgr2=0x7f385c0faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 -- 172.21.15.45:0/2378886282 shutdown_connections 2024-04-16T11:33:28.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.571+0000 7f3863a00700 1 -- 172.21.15.45:0/2378886282 wait complete. 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.572+0000 7f3863a00700 1 Processor -- start 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.572+0000 7f3863a00700 1 -- start start 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.573+0000 7f3863a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c1153d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.573+0000 7f3863a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c115910 con 0x7f385c070360 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.573+0000 7f3861000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c1153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.573+0000 7f3861000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c1153d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:32872/0 (socket says 172.21.15.45:32872) 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.573+0000 7f3861000700 1 -- 172.21.15.45:0/663870321 learned_addr learned my addr 172.21.15.45:0/663870321 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:28.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.574+0000 7f3861000700 1 -- 172.21.15.45:0/663870321 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850005040 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.574+0000 7f3861000700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c1153d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f385000b7f0 tx=0x7f3850009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.575+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3850009370 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.575+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c115b10 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.575+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c115f80 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.575+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3850013420 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.575+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3850012580 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.576+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c0052b0 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.577+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f38500094d0 con 0x7f385c070360 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.577+0000 7f3858c00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f3844036970 0x7f3844038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.577+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f38500494a0 con 0x7f385c070360 2024-04-16T11:33:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.578+0000 7f385be00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f3844036970 0x7f3844038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.578+0000 7f385be00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f3844036970 0x7f3844038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3848009990 tx=0x7f3848006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:28.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.585+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3850013600 con 0x7f385c070360 2024-04-16T11:33:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.873+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f383c005560 con 0x7f385c070360 2024-04-16T11:33:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.874+0000 7f3858c00700 1 -- 172.21.15.45:0/663870321 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3850011370 con 0x7f385c070360 2024-04-16T11:33:28.874 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:28.874 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f3844036970 msgr2=0x7f3844038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f3844036970 0x7f3844038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3848009990 tx=0x7f3848006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 msgr2=0x7f385c1153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:28.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c1153d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f385000b7f0 tx=0x7f3850009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 shutdown_connections 2024-04-16T11:33:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f3844036970 0x7f3844038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 --2- 172.21.15.45:0/663870321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f385c070360 0x7f385c1153d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 >> 172.21.15.45:0/663870321 conn(0x7f385c0f8600 msgr2=0x7f385c105990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.876+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 shutdown_connections 2024-04-16T11:33:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:28.877+0000 7f3863a00700 1 -- 172.21.15.45:0/663870321 wait complete. 2024-04-16T11:33:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:28 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:33:29.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:28 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:33:29.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:28 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:33:29.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:29.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:29.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:28 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/663870321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:33:30.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:30.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:30.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:30.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:33:30.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T11:33:30.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:33:30.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:30.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:30.978 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:33:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:31 smithi044 ceph-mon[28731]: Deploying daemon crash.smithi045 on smithi045 2024-04-16T11:33:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:33:33.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 -- 172.21.15.45:0/1084013336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 msgr2=0x7fbb3c0fdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:33.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 --2- 172.21.15.45:0/1084013336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c0fdcc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009c60 tx=0x7fbb30004d10 comp rx=0 tx=0).stop 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 -- 172.21.15.45:0/1084013336 shutdown_connections 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 --2- 172.21.15.45:0/1084013336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c0fdcc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 -- 172.21.15.45:0/1084013336 >> 172.21.15.45:0/1084013336 conn(0x7fbb3c072590 msgr2=0x7fbb3c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 -- 172.21.15.45:0/1084013336 shutdown_connections 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.253+0000 7fbb43000700 1 -- 172.21.15.45:0/1084013336 wait complete. 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.254+0000 7fbb43000700 1 Processor -- start 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.254+0000 7fbb43000700 1 -- start start 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.254+0000 7fbb43000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c1152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:33.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.254+0000 7fbb43000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1157e0 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.255+0000 7fbb41c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c1152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:33.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.255+0000 7fbb41c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c1152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:58142/0 (socket says 172.21.15.45:58142) 2024-04-16T11:33:33.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.255+0000 7fbb41c00700 1 -- 172.21.15.45:0/550587950 learned_addr learned my addr 172.21.15.45:0/550587950 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:33.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.256+0000 7fbb41c00700 1 -- 172.21.15.45:0/550587950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30009940 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.256+0000 7fbb41c00700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c1152a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb300096d0 tx=0x7fbb30004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:33.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.256+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb30007c00 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.256+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c1159e0 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.257+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c115e50 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.257+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb3001d910 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.257+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb300128b0 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.258+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c065920 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.258+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fbb30019040 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.259+0000 7fbb39600700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbb240369c0 0x7fbb24038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:33.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.259+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fbb30049cd0 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.259+0000 7fbb41200700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbb240369c0 0x7fbb24038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:33.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.260+0000 7fbb41200700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbb240369c0 0x7fbb24038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009990 tx=0x7fbb28006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:33.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.274+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbb3004f020 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.597+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb3c05f5f0 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.599+0000 7fbb39600700 1 -- 172.21.15.45:0/550587950 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbb30021020 con 0x7fbb3c0fd8f0 2024-04-16T11:33:33.597 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:33.597 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:33.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbb240369c0 msgr2=0x7fbb24038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:33.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbb240369c0 0x7fbb24038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009990 tx=0x7fbb28006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:33.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 msgr2=0x7fbb3c1152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:33.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c1152a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb300096d0 tx=0x7fbb30004d10 comp rx=0 tx=0).stop 2024-04-16T11:33:33.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 shutdown_connections 2024-04-16T11:33:33.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbb240369c0 0x7fbb24038e70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:33.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 --2- 172.21.15.45:0/550587950 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb3c0fd8f0 0x7fbb3c1152a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:33.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 >> 172.21.15.45:0/550587950 conn(0x7fbb3c072590 msgr2=0x7fbb3c103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:33.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 shutdown_connections 2024-04-16T11:33:33.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:33.601+0000 7fbb43000700 1 -- 172.21.15.45:0/550587950 wait complete. 2024-04-16T11:33:33.602 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:34 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:34.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:34 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:34.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:34 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:34.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:34 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:34.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:34 smithi044 ceph-mon[28731]: Deploying daemon node-exporter.smithi045 on smithi045 2024-04-16T11:33:34.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:34 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/550587950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:35.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:35.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:35.720 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:33:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:36 smithi044 ceph-mon[28731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 -- 172.21.15.45:0/2582772337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 msgr2=0x7fb54c0fded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 --2- 172.21.15.45:0/2582772337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c0fded0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5400048c0 tx=0x7fb540009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:39.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 -- 172.21.15.45:0/2582772337 shutdown_connections 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 --2- 172.21.15.45:0/2582772337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c0fded0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 -- 172.21.15.45:0/2582772337 >> 172.21.15.45:0/2582772337 conn(0x7fb54c071860 msgr2=0x7fb54c071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 -- 172.21.15.45:0/2582772337 shutdown_connections 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.123+0000 7fb554000700 1 -- 172.21.15.45:0/2582772337 wait complete. 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.124+0000 7fb554000700 1 Processor -- start 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.124+0000 7fb554000700 1 -- start start 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.125+0000 7fb554000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c1153e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.125+0000 7fb554000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c115920 con 0x7fb54c0fdb00 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.125+0000 7fb551600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c1153e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.125+0000 7fb551600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c1153e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:58184/0 (socket says 172.21.15.45:58184) 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.125+0000 7fb551600700 1 -- 172.21.15.45:0/594486678 learned_addr learned my addr 172.21.15.45:0/594486678 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.126+0000 7fb551600700 1 -- 172.21.15.45:0/594486678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540005040 con 0x7fb54c0fdb00 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.127+0000 7fb551600700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c1153e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb54000b7f0 tx=0x7fb540009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:39.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.127+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb540009370 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.127+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c115b20 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.127+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb540013420 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.127+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb54001c590 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.127+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c115f90 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.128+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c065900 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.129+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fb540013a70 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.129+0000 7fb549600700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb5340369c0 0x7fb534038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.129+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb5400494c0 con 0x7fb54c0fdb00 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.130+0000 7fb550c00700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb5340369c0 0x7fb534038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:39.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.132+0000 7fb550c00700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb5340369c0 0x7fb534038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb538009990 tx=0x7fb538006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:39.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.144+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb540024d30 con 0x7fb54c0fdb00 2024-04-16T11:33:39.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:38 smithi044 ceph-mon[28731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:39.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.449+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb54c05f5f0 con 0x7fb54c0fdb00 2024-04-16T11:33:39.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.450+0000 7fb549600700 1 -- 172.21.15.45:0/594486678 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb540021070 con 0x7fb54c0fdb00 2024-04-16T11:33:39.450 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:39.450 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb5340369c0 msgr2=0x7fb534038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb5340369c0 0x7fb534038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb538009990 tx=0x7fb538006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 msgr2=0x7fb54c1153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c1153e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb54000b7f0 tx=0x7fb540009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 shutdown_connections 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb5340369c0 0x7fb534038e70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 --2- 172.21.15.45:0/594486678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb54c0fdb00 0x7fb54c1153e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 >> 172.21.15.45:0/594486678 conn(0x7fb54c071860 msgr2=0x7fb54c0faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 shutdown_connections 2024-04-16T11:33:39.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:39.453+0000 7fb554000700 1 -- 172.21.15.45:0/594486678 wait complete. 2024-04-16T11:33:39.453 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:40.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/594486678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:40.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:33:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:39 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:33:41.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:41.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:40 smithi044 ceph-mon[28731]: Deploying daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:33:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:40 smithi044 ceph-mon[28731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:41.664 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:33:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:43.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:42 smithi044 ceph-mon[28731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:43.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 -- 172.21.15.45:0/1276577505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 msgr2=0x7ff97c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 --2- 172.21.15.45:0/1276577505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c070730 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0048c0 tx=0x7ff96c009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 -- 172.21.15.45:0/1276577505 shutdown_connections 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 --2- 172.21.15.45:0/1276577505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c070730 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 -- 172.21.15.45:0/1276577505 >> 172.21.15.45:0/1276577505 conn(0x7ff97c0f8600 msgr2=0x7ff97c0faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 -- 172.21.15.45:0/1276577505 shutdown_connections 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.988+0000 7ff982c00700 1 -- 172.21.15.45:0/1276577505 wait complete. 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.989+0000 7ff982c00700 1 Processor -- start 2024-04-16T11:33:43.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.989+0000 7ff982c00700 1 -- start start 2024-04-16T11:33:43.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.989+0000 7ff982c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c1153c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:43.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.989+0000 7ff982c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c115900 con 0x7ff97c070360 2024-04-16T11:33:43.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.990+0000 7ff97be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c1153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:43.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.990+0000 7ff97be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c1153c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.45:39542/0 (socket says 172.21.15.45:39542) 2024-04-16T11:33:43.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.990+0000 7ff97be00700 1 -- 172.21.15.45:0/2185136458 learned_addr learned my addr 172.21.15.45:0/2185136458 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:43.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.991+0000 7ff97be00700 1 -- 172.21.15.45:0/2185136458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c005040 con 0x7ff97c070360 2024-04-16T11:33:43.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.991+0000 7ff97be00700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c1153c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00b7f0 tx=0x7ff96c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:43.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.992+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c009370 con 0x7ff97c070360 2024-04-16T11:33:43.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.992+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff96c013420 con 0x7ff97c070360 2024-04-16T11:33:43.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.992+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c01c590 con 0x7ff97c070360 2024-04-16T11:33:43.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.992+0000 7ff982c00700 1 -- 172.21.15.45:0/2185136458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c115b00 con 0x7ff97c070360 2024-04-16T11:33:43.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.992+0000 7ff982c00700 1 -- 172.21.15.45:0/2185136458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c115f70 con 0x7ff97c070360 2024-04-16T11:33:43.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.993+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9540052b0 con 0x7ff97c070360 2024-04-16T11:33:43.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.994+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7ff96c013a70 con 0x7ff97c070360 2024-04-16T11:33:43.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.994+0000 7ff973e00700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff960036970 0x7ff960038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:43.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.994+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff96c0494c0 con 0x7ff97c070360 2024-04-16T11:33:43.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.995+0000 7ff97b400700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff960036970 0x7ff960038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:43.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:43.996+0000 7ff97b400700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff960036970 0x7ff960038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff968009990 tx=0x7ff968006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:44.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.011+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff96c024d30 con 0x7ff97c070360 2024-04-16T11:33:44.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.305+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff954005560 con 0x7ff97c070360 2024-04-16T11:33:44.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.306+0000 7ff973e00700 1 -- 172.21.15.45:0/2185136458 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff96c021070 con 0x7ff97c070360 2024-04-16T11:33:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:44.304 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:30:19.424189Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T11:33:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff960036970 msgr2=0x7ff960038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff960036970 0x7ff960038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff968009990 tx=0x7ff968006e40 comp rx=0 tx=0).stop 2024-04-16T11:33:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 msgr2=0x7ff97c1153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c1153c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00b7f0 tx=0x7ff96c009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 shutdown_connections 2024-04-16T11:33:44.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff960036970 0x7ff960038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:44.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 --2- 172.21.15.45:0/2185136458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff97c070360 0x7ff97c1153c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:44.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 >> 172.21.15.45:0/2185136458 conn(0x7ff97c0f8600 msgr2=0x7ff97c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:44.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 shutdown_connections 2024-04-16T11:33:44.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:44.309+0000 7ff971600700 1 -- 172.21.15.45:0/2185136458 wait complete. 2024-04-16T11:33:44.308 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-04-16T11:33:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: Deploying daemon mon.smithi045 on smithi045 2024-04-16T11:33:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:44 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2185136458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:45.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T11:33:45.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T11:33:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:33:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:33:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:46 smithi044 ceph-mon[28731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:48 smithi044 ceph-mon[28731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:54.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:33:54.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: mon.smithi044 calling monitor election 2024-04-16T11:33:54.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:54.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: mon.smithi045 calling monitor election 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/1570465555' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: mon.smithi044 is new leader, mons smithi044,smithi045 in quorum (ranks 0,1) 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: monmap e2: 2 mons at {smithi044=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: fsmap 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: mgrmap e17: smithi044.sczobv(active, since 38s) 2024-04-16T11:33:54.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: overall HEALTH_OK 2024-04-16T11:33:54.729 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:33:54.729 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/1570465555' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:33:54.729 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:54.729 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/1570465555' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:33:54.729 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/1570465555' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:33:54.729 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:54.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 -- 172.21.15.45:0/2534737397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e440061f0 msgr2=0x7f5e44008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 --2- 172.21.15.45:0/2534737397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e440061f0 0x7f5e44008650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0657c0 tx=0x7f5e50002040 comp rx=0 tx=0).stop 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 -- 172.21.15.45:0/2534737397 shutdown_connections 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 --2- 172.21.15.45:0/2534737397 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e440061f0 0x7f5e44008650 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 -- 172.21.15.45:0/2534737397 >> 172.21.15.45:0/2534737397 conn(0x7f5e5c0719d0 msgr2=0x7f5e5c071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 -- 172.21.15.45:0/2534737397 shutdown_connections 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.561+0000 7f5e63600700 1 -- 172.21.15.45:0/2534737397 wait complete. 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.562+0000 7f5e63600700 1 Processor -- start 2024-04-16T11:33:55.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.563+0000 7f5e63600700 1 -- start start 2024-04-16T11:33:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.563+0000 7f5e63600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.563+0000 7f5e63600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5c115fb0 0x7f5e5c10f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.563+0000 7f5e63600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c10fd20 con 0x7f5e5c115fb0 2024-04-16T11:33:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.563+0000 7f5e63600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c10fe90 con 0x7f5e5c1156a0 2024-04-16T11:33:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.563+0000 7f5e60c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e60c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36242/0 (socket says 172.21.15.45:36242) 2024-04-16T11:33:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e60c00700 1 -- 172.21.15.45:0/4212311382 learned_addr learned my addr 172.21.15.45:0/4212311382 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:33:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e5be00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5c115fb0 0x7f5e5c10f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e60c00700 1 -- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 msgr2=0x7f5e5c115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T11:33:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e60c00700 1 -- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 msgr2=0x7f5e5c115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e60c00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e60c00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e5be00700 1 -- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 msgr2=0x7f5e5c115a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e5be00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.564+0000 7f5e5be00700 1 -- 172.21.15.45:0/4212311382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50005040 con 0x7f5e5c115fb0 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.565+0000 7f5e5be00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5c115fb0 0x7f5e5c10f640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800b790 tx=0x7f5e480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.565+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e48011810 con 0x7f5e5c115fb0 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.565+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c110170 con 0x7f5e5c115fb0 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.565+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e48011e40 con 0x7f5e5c115fb0 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.565+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4800f2e0 con 0x7f5e5c115fb0 2024-04-16T11:33:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.565+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c11a6c0 con 0x7f5e5c115fb0 2024-04-16T11:33:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.568+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c065e40 con 0x7f5e5c115fb0 2024-04-16T11:33:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.568+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f5e48010380 con 0x7f5e5c115fb0 2024-04-16T11:33:55.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.569+0000 7f5e58c00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f5e3c069360 0x7f5e3c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.569+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f5e48086360 con 0x7f5e5c115fb0 2024-04-16T11:33:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.570+0000 7f5e60c00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f5e3c069360 0x7f5e3c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.571+0000 7f5e60c00700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f5e3c069360 0x7f5e3c06b810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5e500045a0 tx=0x7f5e50002040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:55.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.585+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5e48057180 con 0x7f5e5c115fb0 2024-04-16T11:33:55.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: mgrmap e18: smithi044.sczobv(active, since 38s), standbys: smithi045.sdlwif 2024-04-16T11:33:55.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:33:55.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:33:55.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: mgrmap e18: smithi044.sczobv(active, since 38s), standbys: smithi045.sdlwif 2024-04-16T11:33:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:33:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:33:55.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:33:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.897+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e5c1048f0 con 0x7f5e5c115fb0 2024-04-16T11:33:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.898+0000 7f5e58c00700 1 -- 172.21.15.45:0/4212311382 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7f5e48015090 con 0x7f5e5c115fb0 2024-04-16T11:33:55.896 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-16T11:33:55.897 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","modified":"2024-04-16T11:33:49.284289Z","created":"2024-04-16T11:30:19.424189Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi044","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-16T11:33:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.901+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f5e3c069360 msgr2=0x7f5e3c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.901+0000 7f5e63600700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f5e3c069360 0x7f5e3c06b810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5e500045a0 tx=0x7f5e50002040 comp rx=0 tx=0).stop 2024-04-16T11:33:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.901+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5c115fb0 msgr2=0x7f5e5c10f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.901+0000 7f5e63600700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5c115fb0 0x7f5e5c10f640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800b790 tx=0x7f5e480091a0 comp rx=0 tx=0).stop 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 shutdown_connections 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f5e3c069360 0x7f5e3c06b810 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e5c1156a0 0x7f5e5c115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 --2- 172.21.15.45:0/4212311382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e5c115fb0 0x7f5e5c10f640 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 >> 172.21.15.45:0/4212311382 conn(0x7f5e5c0719d0 msgr2=0x7f5e5c102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 shutdown_connections 2024-04-16T11:33:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:33:55.902+0000 7f5e63600700 1 -- 172.21.15.45:0/4212311382 wait complete. 2024-04-16T11:33:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 2 2024-04-16T11:33:56.388 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:56 smithi045 ceph-mon[36308]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:56.388 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:56 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/4212311382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:56.388 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:56.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:56 smithi044 ceph-mon[28731]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:56.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:56 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/4212311382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T11:33:56.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:56.900 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-16T11:33:56.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-16T11:33:57.393 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:33:58.966 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:58.966 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:58.967 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:33:58.967 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:33:58.967 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:33:58.967 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:33:58.967 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:58 smithi044 ceph-mon[28731]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:33:59.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:58 smithi045 ceph-mon[36308]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:33:59.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 -- 172.21.15.44:0/2035094453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 msgr2=0x7f83e0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:59.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 --2- 172.21.15.44:0/2035094453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e0102ea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f83d00048c0 tx=0x7f83d0009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:59.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 -- 172.21.15.44:0/2035094453 shutdown_connections 2024-04-16T11:33:59.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 --2- 172.21.15.44:0/2035094453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e0102ea0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:59.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 -- 172.21.15.44:0/2035094453 >> 172.21.15.44:0/2035094453 conn(0x7f83e00f85f0 msgr2=0x7f83e00faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:59.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 -- 172.21.15.44:0/2035094453 shutdown_connections 2024-04-16T11:33:59.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.509+0000 7f83e6e00700 1 -- 172.21.15.44:0/2035094453 wait complete. 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.510+0000 7f83e6e00700 1 Processor -- start 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83e6e00700 1 -- start start 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83e6e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83e6e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83e0070800 0x7f83e0070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83e6e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0119540 con 0x7f83e0102ad0 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83e6e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0119680 con 0x7f83e0070800 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83dfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:59.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83dfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56164/0 (socket says 172.21.15.44:56164) 2024-04-16T11:33:59.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.511+0000 7f83dfe00700 1 -- 172.21.15.44:0/596818766 learned_addr learned my addr 172.21.15.44:0/596818766 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.512+0000 7f83dfe00700 1 -- 172.21.15.44:0/596818766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83e0070800 msgr2=0x7f83e0070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.512+0000 7f83df400700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83e0070800 0x7f83e0070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.512+0000 7f83dfe00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83e0070800 0x7f83e0070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.512+0000 7f83dfe00700 1 -- 172.21.15.44:0/596818766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0005040 con 0x7f83e0102ad0 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.512+0000 7f83dfe00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e00702c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f83d000b5c0 tx=0x7f83d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.513+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d000e040 con 0x7f83e0102ad0 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.513+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83d0009950 con 0x7f83e0102ad0 2024-04-16T11:33:59.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.513+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e0119820 con 0x7f83e0102ad0 2024-04-16T11:33:59.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.513+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d001c3f0 con 0x7f83e0102ad0 2024-04-16T11:33:59.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.513+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e0119de0 con 0x7f83e0102ad0 2024-04-16T11:33:59.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.514+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e0101d40 con 0x7f83e0102ad0 2024-04-16T11:33:59.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.515+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f83d00137d0 con 0x7f83e0102ad0 2024-04-16T11:33:59.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.516+0000 7f83d7e00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f83c4069430 0x7f83c406b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:33:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.516+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f83d0087400 con 0x7f83e0102ad0 2024-04-16T11:33:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.517+0000 7f83df400700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f83c4069430 0x7f83c406b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:33:59.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.517+0000 7f83df400700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f83c4069430 0x7f83c406b8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f83cc006fd0 tx=0x7f83cc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:33:59.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.521+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f83d00547f0 con 0x7f83e0102ad0 2024-04-16T11:33:59.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.743+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f83e00fe1d0 con 0x7f83e0102ad0 2024-04-16T11:33:59.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.744+0000 7f83d7e00700 1 -- 172.21.15.44:0/596818766 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f83d0057e10 con 0x7f83e0102ad0 2024-04-16T11:33:59.743 INFO:teuthology.orchestra.run.smithi044.stdout:# minimal ceph.conf for 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:33:59.743 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-16T11:33:59.743 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:33:59.743 INFO:teuthology.orchestra.run.smithi044.stdout: mon_host = [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-16T11:33:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.747+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f83c4069430 msgr2=0x7f83c406b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.747+0000 7f83e6e00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f83c4069430 0x7f83c406b8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f83cc006fd0 tx=0x7f83cc008040 comp rx=0 tx=0).stop 2024-04-16T11:33:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.747+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 msgr2=0x7f83e00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:33:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.747+0000 7f83e6e00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e00702c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f83d000b5c0 tx=0x7f83d0009e30 comp rx=0 tx=0).stop 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 shutdown_connections 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f83c4069430 0x7f83c406b8e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f83e0102ad0 0x7f83e00702c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 --2- 172.21.15.44:0/596818766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83e0070800 0x7f83e0070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 >> 172.21.15.44:0/596818766 conn(0x7f83e00f85f0 msgr2=0x7f83e00f9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 shutdown_connections 2024-04-16T11:33:59.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:33:59.748+0000 7f83e6e00700 1 -- 172.21.15.44:0/596818766 wait complete. 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:00.054 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:33:59 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/596818766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:34:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:34:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:33:59 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/596818766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:00.424 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-16T11:34:00.424 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:34:00.424 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T11:34:00.527 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:34:00.527 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:34:00.681 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:34:00.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T11:34:00.796 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:34:00.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:34:00.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:00 smithi044 ceph-mon[28731]: Reconfiguring mon.smithi044 (unknown last config time)... 2024-04-16T11:34:00.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:00 smithi044 ceph-mon[28731]: Reconfiguring daemon mon.smithi044 on smithi044 2024-04-16T11:34:00.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:00 smithi044 ceph-mon[28731]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:00.940 INFO:tasks.cephadm:Deploying OSDs... 2024-04-16T11:34:00.940 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:34:00.941 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T11:34:00.998 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T11:34:00.998 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-04-16T11:34:01.055 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-16 11:33:04.826414316 +0000 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-16 11:25:37.219467958 +0000 2024-04-16T11:34:01.056 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-16 11:25:37.219467958 +0000 2024-04-16T11:34:01.057 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-16 11:25:37.219467958 +0000 2024-04-16T11:34:01.057 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T11:34:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:00 smithi045 ceph-mon[36308]: Reconfiguring mon.smithi044 (unknown last config time)... 2024-04-16T11:34:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:00 smithi045 ceph-mon[36308]: Reconfiguring daemon mon.smithi044 on smithi044 2024-04-16T11:34:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:00 smithi045 ceph-mon[36308]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-16T11:34:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-16T11:34:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.00034701 s, 1.5 MB/s 2024-04-16T11:34:01.156 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T11:34:01.218 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-04-16T11:34:01.265 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T11:34:01.265 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-16 11:33:04.826414316 +0000 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-16 11:25:37.873457953 +0000 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-16 11:25:37.873457953 +0000 2024-04-16T11:34:01.266 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-16 11:25:37.873457953 +0000 2024-04-16T11:34:01.266 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T11:34:01.349 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-16T11:34:01.349 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-16T11:34:01.349 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000316551 s, 1.6 MB/s 2024-04-16T11:34:01.352 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T11:34:01.406 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-04-16T11:34:01.459 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T11:34:01.459 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-16 11:33:04.827414301 +0000 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-16 11:25:38.494448452 +0000 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-16 11:25:38.494448452 +0000 2024-04-16T11:34:01.460 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-16 11:25:38.494448452 +0000 2024-04-16T11:34:01.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T11:34:01.541 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-16T11:34:01.542 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-16T11:34:01.542 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000355792 s, 1.4 MB/s 2024-04-16T11:34:01.546 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T11:34:01.602 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-04-16T11:34:01.653 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T11:34:01.653 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:01.653 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-16T11:34:01.653 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:01.653 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:01.654 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-16 11:33:04.828414286 +0000 2024-04-16T11:34:01.654 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-16 11:25:39.126438784 +0000 2024-04-16T11:34:01.654 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-16 11:25:39.126438784 +0000 2024-04-16T11:34:01.654 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-16 11:25:39.126438784 +0000 2024-04-16T11:34:01.654 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T11:34:01.738 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-16T11:34:01.738 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-16T11:34:01.738 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000295551 s, 1.7 MB/s 2024-04-16T11:34:01.741 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T11:34:01.799 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:34:01.799 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T11:34:01.860 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T11:34:01.860 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-04-16T11:34:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T11:34:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:01.913 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-16T11:34:01.913 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:01.914 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:01.914 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 11:33:47.776644653 +0000 2024-04-16T11:34:01.914 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 11:25:39.688441729 +0000 2024-04-16T11:34:01.914 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 11:25:39.688441729 +0000 2024-04-16T11:34:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 11:25:39.688441729 +0000 2024-04-16T11:34:01.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T11:34:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:02.009 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T11:34:02.009 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T11:34:02.009 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000373747 s, 1.4 MB/s 2024-04-16T11:34:02.015 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T11:34:02.073 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-04-16T11:34:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:02.126 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 11:33:47.777644637 +0000 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 11:25:40.305432414 +0000 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 11:25:40.305432414 +0000 2024-04-16T11:34:02.127 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 11:25:40.305432414 +0000 2024-04-16T11:34:02.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T11:34:02.264 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T11:34:02.264 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T11:34:02.264 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000257538 s, 2.0 MB/s 2024-04-16T11:34:02.268 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T11:34:02.328 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-04-16T11:34:02.380 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T11:34:02.380 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 11:33:47.777644637 +0000 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 11:25:41.007421817 +0000 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 11:25:41.007421817 +0000 2024-04-16T11:34:02.381 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 11:25:41.007421817 +0000 2024-04-16T11:34:02.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T11:34:02.465 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T11:34:02.466 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T11:34:02.466 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000243486 s, 2.1 MB/s 2024-04-16T11:34:02.468 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T11:34:02.522 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-04-16T11:34:02.571 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-16 11:33:47.778644622 +0000 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-16 11:25:41.632412381 +0000 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-16 11:25:41.632412381 +0000 2024-04-16T11:34:02.572 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-16 11:25:41.632412381 +0000 2024-04-16T11:34:02.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T11:34:02.663 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-16T11:34:02.664 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-16T11:34:02.664 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000227953 s, 2.2 MB/s 2024-04-16T11:34:02.668 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T11:34:02.729 INFO:tasks.cephadm:Deploying osd.0 on smithi044 with /dev/vg_nvme/lv_4... 2024-04-16T11:34:02.729 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T11:34:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:02 smithi045 ceph-mon[36308]: Reconfiguring mgr.smithi044.sczobv (unknown last config time)... 2024-04-16T11:34:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:02 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:34:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:02 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:34:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:02 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:03.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:02 smithi045 ceph-mon[36308]: Reconfiguring daemon mgr.smithi044.sczobv on smithi044 2024-04-16T11:34:03.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:02 smithi045 ceph-mon[36308]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:03.203 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:02 smithi044 ceph-mon[28731]: Reconfiguring mgr.smithi044.sczobv (unknown last config time)... 2024-04-16T11:34:03.204 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:02 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:34:03.204 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:02 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:34:03.204 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:02 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:03.204 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:02 smithi044 ceph-mon[28731]: Reconfiguring daemon mgr.smithi044.sczobv on smithi044 2024-04-16T11:34:03.204 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:02 smithi044 ceph-mon[28731]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:03.639 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:34:04.840 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:04.840 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:04.840 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: Reconfiguring crash.smithi044 (monmap changed)... 2024-04-16T11:34:04.840 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:34:04.840 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:04.840 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: Reconfiguring daemon crash.smithi044 on smithi044 2024-04-16T11:34:04.841 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:04 smithi044 ceph-mon[28731]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: Reconfiguring crash.smithi044 (monmap changed)... 2024-04-16T11:34:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:34:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:05.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: Reconfiguring daemon crash.smithi044 on smithi044 2024-04-16T11:34:05.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:04 smithi045 ceph-mon[36308]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T11:34:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T11:34:06.552 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-16T11:34:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-16T11:34:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0351815 s, 298 MB/s 2024-04-16T11:34:06.552 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-16T11:34:07.165 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:06 smithi044 ceph-mon[28731]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:07.165 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:07.165 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:06 smithi045 ceph-mon[36308]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:07.567 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_4 2024-04-16T11:34:08.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:07 smithi044 ceph-mon[28731]: Reconfiguring alertmanager.smithi044 (dependencies changed)... 2024-04-16T11:34:08.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:07 smithi044 ceph-mon[28731]: Reconfiguring daemon alertmanager.smithi044 on smithi044 2024-04-16T11:34:08.209 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:07 smithi045 ceph-mon[36308]: Reconfiguring alertmanager.smithi044 (dependencies changed)... 2024-04-16T11:34:08.209 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:07 smithi045 ceph-mon[36308]: Reconfiguring daemon alertmanager.smithi044 on smithi044 2024-04-16T11:34:09.208 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:08 smithi044 ceph-mon[28731]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:08 smithi045 ceph-mon[36308]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:09.346 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:34:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:10 smithi044 ceph-mon[28731]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:10 smithi045 ceph-mon[36308]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:13.061 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:12 smithi044 ceph-mon[28731]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:13.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:12 smithi045 ceph-mon[36308]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:13.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 -- 172.21.15.44:0/3159059496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fcc30 msgr2=0x7fba740fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:34:13.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 --2- 172.21.15.44:0/3159059496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fcc30 0x7fba740fd040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba64004940 tx=0x7fba64009e30 comp rx=0 tx=0).stop 2024-04-16T11:34:13.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 -- 172.21.15.44:0/3159059496 shutdown_connections 2024-04-16T11:34:13.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 --2- 172.21.15.44:0/3159059496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fdee0 0x7fba740fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:34:13.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 --2- 172.21.15.44:0/3159059496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fcc30 0x7fba740fd040 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:34:13.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 -- 172.21.15.44:0/3159059496 >> 172.21.15.44:0/3159059496 conn(0x7fba740f83d0 msgr2=0x7fba740fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:34:13.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 -- 172.21.15.44:0/3159059496 shutdown_connections 2024-04-16T11:34:13.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.715+0000 7fba73200700 1 -- 172.21.15.44:0/3159059496 wait complete. 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.716+0000 7fba73200700 1 Processor -- start 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.716+0000 7fba73200700 1 -- start start 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba73200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fcc30 0x7fba7410f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba73200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 0x7fba7410ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba73200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74110510 con 0x7fba740fdee0 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba73200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74110d60 con 0x7fba740fcc30 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba71400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 0x7fba7410ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba71400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 0x7fba7410ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41696/0 (socket says 172.21.15.44:41696) 2024-04-16T11:34:13.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba71400700 1 -- 172.21.15.44:0/942885270 learned_addr learned my addr 172.21.15.44:0/942885270 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:34:13.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.717+0000 7fba71e00700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fcc30 0x7fba7410f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:34:13.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba71400700 1 -- 172.21.15.44:0/942885270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fcc30 msgr2=0x7fba7410f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:34:13.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba71400700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fcc30 0x7fba7410f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:34:13.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba71400700 1 -- 172.21.15.44:0/942885270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64005040 con 0x7fba740fdee0 2024-04-16T11:34:13.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba71e00700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fcc30 0x7fba7410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:34:13.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba71400700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 0x7fba7410ff10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba6000eb40 tx=0x7fba6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:34:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba60007cc0 con 0x7fba740fdee0 2024-04-16T11:34:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba6000f040 con 0x7fba740fdee0 2024-04-16T11:34:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.718+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba74110fe0 con 0x7fba740fdee0 2024-04-16T11:34:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.719+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba60009620 con 0x7fba740fdee0 2024-04-16T11:34:13.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.719+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba74111520 con 0x7fba740fdee0 2024-04-16T11:34:13.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.721+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba74101870 con 0x7fba740fdee0 2024-04-16T11:34:13.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.721+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fba60009780 con 0x7fba740fdee0 2024-04-16T11:34:13.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.722+0000 7fba6a000700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fba58069210 0x7fba5806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:34:13.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.722+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fba60085c90 con 0x7fba740fdee0 2024-04-16T11:34:13.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.722+0000 7fba71e00700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fba58069210 0x7fba5806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:34:13.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.723+0000 7fba71e00700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fba58069210 0x7fba5806b6c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b0d0 tx=0x7fba64009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:34:13.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.727+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fba6004e250 con 0x7fba740fdee0 2024-04-16T11:34:13.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:13.989+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fba74105890 con 0x7fba58069210 2024-04-16T11:34:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:14 smithi044 ceph-mon[28731]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:14 smithi044 ceph-mon[28731]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:34:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:14 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:34:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:14 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:34:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:14 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:14 smithi045 ceph-mon[36308]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:14 smithi045 ceph-mon[36308]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:34:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:14 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:34:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:14 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:34:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:14 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:16.424 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:16.424 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:16.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:17.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:17 smithi044 ceph-mon[28731]: Reconfiguring grafana.smithi044 (dependencies changed)... 2024-04-16T11:34:17.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:17 smithi044 ceph-mon[28731]: Reconfiguring daemon grafana.smithi044 on smithi044 2024-04-16T11:34:17.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:17 smithi044 ceph-mon[28731]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:17.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:17 smithi045 ceph-mon[36308]: Reconfiguring grafana.smithi044 (dependencies changed)... 2024-04-16T11:34:17.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:17 smithi045 ceph-mon[36308]: Reconfiguring daemon grafana.smithi044 on smithi044 2024-04-16T11:34:17.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:17 smithi045 ceph-mon[36308]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:18.455 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:18 smithi044 ceph-mon[28731]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:18.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:18 smithi045 ceph-mon[36308]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:21.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:20 smithi044 ceph-mon[28731]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:21.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:20 smithi045 ceph-mon[36308]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:22 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:22 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:22 smithi045 ceph-mon[36308]: Reconfiguring prometheus.smithi044 (dependencies changed)... 2024-04-16T11:34:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:22 smithi045 ceph-mon[36308]: Reconfiguring daemon prometheus.smithi044 on smithi044 2024-04-16T11:34:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:22 smithi045 ceph-mon[36308]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:22 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:22 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:22 smithi044 ceph-mon[28731]: Reconfiguring prometheus.smithi044 (dependencies changed)... 2024-04-16T11:34:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:22 smithi044 ceph-mon[28731]: Reconfiguring daemon prometheus.smithi044 on smithi044 2024-04-16T11:34:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:22 smithi044 ceph-mon[28731]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:24.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:23 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2365208924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f85ef5f-daca-46b0-a83b-2fab671b920a"}]: dispatch 2024-04-16T11:34:24.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:23 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2365208924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f85ef5f-daca-46b0-a83b-2fab671b920a"}]': finished 2024-04-16T11:34:24.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:23 smithi045 ceph-mon[36308]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T11:34:24.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:23 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:24.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:23 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2365208924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f85ef5f-daca-46b0-a83b-2fab671b920a"}]: dispatch 2024-04-16T11:34:24.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:23 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2365208924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f85ef5f-daca-46b0-a83b-2fab671b920a"}]': finished 2024-04-16T11:34:24.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:23 smithi044 ceph-mon[28731]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T11:34:24.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:23 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:25.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:24 smithi044 ceph-mon[28731]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:25.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:24 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1868543193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:34:25.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:24 smithi045 ceph-mon[36308]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:25.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:24 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1868543193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:34:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:26 smithi044 ceph-mon[28731]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:26 smithi045 ceph-mon[36308]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:28 smithi044 ceph-mon[28731]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:29.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:28 smithi045 ceph-mon[36308]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:30 smithi044 ceph-mon[28731]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:30 smithi045 ceph-mon[36308]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:30 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:33.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:32 smithi044 ceph-mon[28731]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:32 smithi045 ceph-mon[36308]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:35.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:34 smithi044 ceph-mon[28731]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:35.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:34 smithi045 ceph-mon[36308]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:37.071 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:36 smithi044 ceph-mon[28731]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:37.071 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:37.071 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:37.071 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:34:37.071 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:37.326 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:36 smithi045 ceph-mon[36308]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:37.327 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:37.327 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:37.327 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:34:37.327 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:38.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:37 smithi045 ceph-mon[36308]: Reconfiguring crash.smithi045 (monmap changed)... 2024-04-16T11:34:38.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:37 smithi045 ceph-mon[36308]: Reconfiguring daemon crash.smithi045 on smithi045 2024-04-16T11:34:38.422 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:37 smithi044 ceph-mon[28731]: Reconfiguring crash.smithi045 (monmap changed)... 2024-04-16T11:34:38.422 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:37 smithi044 ceph-mon[28731]: Reconfiguring daemon crash.smithi045 on smithi045 2024-04-16T11:34:39.266 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:39.266 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T11:34:39.267 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:39.267 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:39.267 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:39.267 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:34:39.267 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:34:39.267 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:39.295 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:39.296 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T11:34:39.296 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:39.296 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:39.296 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:39.297 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:34:39.297 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:34:39.298 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:40.327 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:40 smithi044 ceph-mon[28731]: Deploying daemon osd.0 on smithi044 2024-04-16T11:34:40.328 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:40 smithi044 ceph-mon[28731]: Reconfiguring mgr.smithi045.sdlwif (monmap changed)... 2024-04-16T11:34:40.328 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:40 smithi044 ceph-mon[28731]: Reconfiguring daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:34:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:40 smithi045 ceph-mon[36308]: Deploying daemon osd.0 on smithi044 2024-04-16T11:34:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:40 smithi045 ceph-mon[36308]: Reconfiguring mgr.smithi045.sdlwif (monmap changed)... 2024-04-16T11:34:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:40 smithi045 ceph-mon[36308]: Reconfiguring daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:34:41.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:41 smithi044 ceph-mon[28731]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:41.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:41.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:41.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:34:41.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:34:41.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:41 smithi045 ceph-mon[36308]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:34:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:34:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:42 smithi044 ceph-mon[28731]: Reconfiguring mon.smithi045 (monmap changed)... 2024-04-16T11:34:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:42 smithi044 ceph-mon[28731]: Reconfiguring daemon mon.smithi045 on smithi045 2024-04-16T11:34:42.270 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:42 smithi045 ceph-mon[36308]: Reconfiguring mon.smithi045 (monmap changed)... 2024-04-16T11:34:42.271 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:42 smithi045 ceph-mon[36308]: Reconfiguring daemon mon.smithi045 on smithi045 2024-04-16T11:34:43.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:43.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi044.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi044.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:34:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi044.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi044.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:43.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:34:44.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:44 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:34:44.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:44 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T11:34:44.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:44 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:34:44.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:44 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi044.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T11:34:44.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:44 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:34:44.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:44 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi044.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T11:34:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:44 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:34:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:44 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T11:34:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:44 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:34:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:44 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi044.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T11:34:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:44 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:34:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:44 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi044.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T11:34:45.306 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:45 smithi044 ceph-mon[28731]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:45 smithi045 ceph-mon[36308]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:46.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:46.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:46.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:46 smithi045 ceph-mon[36308]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:46.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:46.638 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:46.639 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:46.639 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:46 smithi044 ceph-mon[28731]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:46.639 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:34:48.880 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:48.880 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:48.880 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:48 smithi045 ceph-mon[36308]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:48.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:48.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:48.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:48 smithi044 ceph-mon[28731]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:51.166 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:50 smithi044 ceph-mon[28731]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:51.166 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:51.167 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:50 smithi045 ceph-mon[36308]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:51 smithi044 ceph-mon[28731]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T11:34:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:51 smithi045 ceph-mon[36308]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T11:34:52.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.792+0000 7fba6a000700 1 -- 172.21.15.44:0/942885270 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fba74105890 con 0x7fba58069210 2024-04-16T11:34:52.792 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 0 on host 'smithi044' 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fba58069210 msgr2=0x7fba5806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fba58069210 0x7fba5806b6c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b0d0 tx=0x7fba64009e30 comp rx=0 tx=0).stop 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 msgr2=0x7fba7410ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 0x7fba7410ff10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba6000eb40 tx=0x7fba6000b040 comp rx=0 tx=0).stop 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 shutdown_connections 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fba58069210 0x7fba5806b6c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:34:52.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba740fcc30 0x7fba7410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:34:52.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 --2- 172.21.15.44:0/942885270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba740fdee0 0x7fba7410ff10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:34:52.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.795+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 >> 172.21.15.44:0/942885270 conn(0x7fba740f83d0 msgr2=0x7fba740f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:34:52.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.796+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 shutdown_connections 2024-04-16T11:34:52.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:34:52.796+0000 7fba73200700 1 -- 172.21.15.44:0/942885270 wait complete. 2024-04-16T11:34:52.945 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:52.945 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T11:34:52.945 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:52.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:52 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:53.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:53.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T11:34:53.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T11:34:53.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:53.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:34:53.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:34:53.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:34:53.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:53.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:34:53.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:53.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:52 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:53.606 DEBUG:teuthology.orchestra.run.smithi044:osd.0> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.0.service 2024-04-16T11:34:53.614 INFO:tasks.cephadm:Deploying osd.1 on smithi044 with /dev/vg_nvme/lv_3... 2024-04-16T11:34:53.614 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T11:34:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:53 smithi044 ceph-mon[28731]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-16T11:34:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:53 smithi044 ceph-mon[28731]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T11:34:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:53 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:53 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:54.242 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:53 smithi045 ceph-mon[36308]: from='osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-16T11:34:54.242 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:53 smithi045 ceph-mon[36308]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T11:34:54.242 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:53 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:54.242 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:53 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:34:55.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:54 smithi044 ceph-mon[28731]: purged_snaps scrub starts 2024-04-16T11:34:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:54 smithi044 ceph-mon[28731]: purged_snaps scrub ok 2024-04-16T11:34:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:54 smithi044 ceph-mon[28731]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:54 smithi044 ceph-mon[28731]: osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] boot 2024-04-16T11:34:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:54 smithi044 ceph-mon[28731]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T11:34:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:54 smithi045 ceph-mon[36308]: purged_snaps scrub starts 2024-04-16T11:34:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:54 smithi045 ceph-mon[36308]: purged_snaps scrub ok 2024-04-16T11:34:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:54 smithi045 ceph-mon[36308]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T11:34:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:54 smithi045 ceph-mon[36308]: osd.0 [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] boot 2024-04-16T11:34:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:54 smithi045 ceph-mon[36308]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T11:34:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:34:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:55 smithi044 ceph-mon[28731]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T11:34:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:55 smithi045 ceph-mon[36308]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T11:34:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:34:57.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:57 smithi045 ceph-mon[36308]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:34:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:57 smithi044 ceph-mon[28731]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:34:57.605 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T11:34:57.605 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T11:34:57.605 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-16T11:34:57.605 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-16T11:34:57.605 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0362583 s, 289 MB/s 2024-04-16T11:34:57.606 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-16T11:34:58.392 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_3 2024-04-16T11:34:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:34:59.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:34:59 smithi045 ceph-mon[36308]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:34:59.381 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:34:59 smithi044 ceph-mon[28731]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:01.276 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:01 smithi044 ceph-mon[28731]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:01.277 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:01.277 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:01.277 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:01 smithi045 ceph-mon[36308]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.804+0000 7fc712000700 1 -- 172.21.15.44:0/1996957427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c1002b0 msgr2=0x7fc70c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.804+0000 7fc712000700 1 --2- 172.21.15.44:0/1996957427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c1002b0 0x7fc70c100720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8004940 tx=0x7fc6f8009e30 comp rx=0 tx=0).stop 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.804+0000 7fc712000700 1 -- 172.21.15.44:0/1996957427 shutdown_connections 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.804+0000 7fc712000700 1 --2- 172.21.15.44:0/1996957427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c1002b0 0x7fc70c100720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.804+0000 7fc712000700 1 --2- 172.21.15.44:0/1996957427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c0ff000 0x7fc70c0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.804+0000 7fc712000700 1 -- 172.21.15.44:0/1996957427 >> 172.21.15.44:0/1996957427 conn(0x7fc70c0fa950 msgr2=0x7fc70c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.805+0000 7fc712000700 1 -- 172.21.15.44:0/1996957427 shutdown_connections 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.805+0000 7fc712000700 1 -- 172.21.15.44:0/1996957427 wait complete. 2024-04-16T11:35:01.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.806+0000 7fc712000700 1 Processor -- start 2024-04-16T11:35:01.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.806+0000 7fc712000700 1 -- start start 2024-04-16T11:35:01.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.806+0000 7fc712000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 0x7fc70c10fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:35:01.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.806+0000 7fc712000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c1002b0 0x7fc70c110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.806+0000 7fc712000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c110650 con 0x7fc70c0ff000 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.806+0000 7fc712000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c1107c0 con 0x7fc70c1002b0 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 0x7fc70c10fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 0x7fc70c10fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33582/0 (socket says 172.21.15.44:33582) 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70b400700 1 -- 172.21.15.44:0/3230819598 learned_addr learned my addr 172.21.15.44:0/3230819598 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70aa00700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c1002b0 0x7fc70c110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70b400700 1 -- 172.21.15.44:0/3230819598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c1002b0 msgr2=0x7fc70c110050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70b400700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c1002b0 0x7fc70c110050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70b400700 1 -- 172.21.15.44:0/3230819598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f8005040 con 0x7fc70c0ff000 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.807+0000 7fc70aa00700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c1002b0 0x7fc70c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.808+0000 7fc70b400700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 0x7fc70c10fb10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00c830 tx=0x7fc6fc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.808+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6fc00a760 con 0x7fc70c0ff000 2024-04-16T11:35:01.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.808+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6fc00ad90 con 0x7fc70c0ff000 2024-04-16T11:35:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.808+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70c115220 con 0x7fc70c0ff000 2024-04-16T11:35:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.808+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70c1157e0 con 0x7fc70c0ff000 2024-04-16T11:35:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.808+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6fc0113a0 con 0x7fc70c0ff000 2024-04-16T11:35:01.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.810+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c04c710 con 0x7fc70c0ff000 2024-04-16T11:35:01.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.811+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fc6fc012400 con 0x7fc70c0ff000 2024-04-16T11:35:01.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.813+0000 7fc703400700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fc6f0069210 0x7fc6f006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:35:01.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.813+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fc6fc086330 con 0x7fc70c0ff000 2024-04-16T11:35:01.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.813+0000 7fc70aa00700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fc6f0069210 0x7fc6f006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:35:01.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.814+0000 7fc70aa00700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fc6f0069210 0x7fc6f006b6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800b7a0 tx=0x7fc6f8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:35:01.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:01.818+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc6fc056f50 con 0x7fc70c0ff000 2024-04-16T11:35:02.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:02.056+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc70c104590 con 0x7fc6f0069210 2024-04-16T11:35:03.275 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:03 smithi044 ceph-mon[28731]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:03.275 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:03 smithi044 ceph-mon[28731]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:35:03.275 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:03 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:35:03.275 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:03 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:35:03.275 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:03 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:03 smithi045 ceph-mon[36308]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:03 smithi045 ceph-mon[36308]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:35:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:03 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:35:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:03 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:35:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:03 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:05.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:05 smithi045 ceph-mon[36308]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:05.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:05 smithi044 ceph-mon[28731]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:07.183 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:07 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:07 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:08 smithi045 ceph-mon[36308]: Detected new or changed devices on smithi044 2024-04-16T11:35:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:08 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi044 to 12870M 2024-04-16T11:35:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:08 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:35:08.394 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:08 smithi044 ceph-mon[28731]: Detected new or changed devices on smithi044 2024-04-16T11:35:08.395 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:08 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi044 to 12870M 2024-04-16T11:35:08.395 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:08 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:35:09.190 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:09 smithi045 ceph-mon[36308]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:09.191 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:09 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/270061402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1"}]: dispatch 2024-04-16T11:35:09.191 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:09 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/270061402' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1"}]': finished 2024-04-16T11:35:09.191 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:09 smithi045 ceph-mon[36308]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T11:35:09.191 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:09 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:09.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:09 smithi044 ceph-mon[28731]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:09.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:09 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/270061402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1"}]: dispatch 2024-04-16T11:35:09.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:09 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/270061402' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1"}]': finished 2024-04-16T11:35:09.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:09 smithi044 ceph-mon[28731]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T11:35:09.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:09 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:10 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2016564461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:35:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:10 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:10 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:10.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:10 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2016564461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:35:10.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:10 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:10.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:10 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:11.452 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:11 smithi044 ceph-mon[28731]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:11.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:11 smithi045 ceph-mon[36308]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:12.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:12 smithi044 ceph-mon[28731]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:12.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:12 smithi045 ceph-mon[36308]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:15.175 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:14 smithi044 ceph-mon[28731]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:14 smithi045 ceph-mon[36308]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:15 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:16.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:16 smithi044 ceph-mon[28731]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:16 smithi045 ceph-mon[36308]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:19.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:18 smithi044 ceph-mon[28731]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:19.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:18 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T11:35:19.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:18 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:18 smithi045 ceph-mon[36308]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:18 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T11:35:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:18 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:19 smithi044 ceph-mon[28731]: Deploying daemon osd.1 on smithi044 2024-04-16T11:35:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:19 smithi045 ceph-mon[36308]: Deploying daemon osd.1 on smithi044 2024-04-16T11:35:21.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:20 smithi044 ceph-mon[28731]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:21.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:20 smithi045 ceph-mon[36308]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:22 smithi044 ceph-mon[28731]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:23.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:22 smithi045 ceph-mon[36308]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:25.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:24 smithi045 ceph-mon[36308]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:25.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:24 smithi044 ceph-mon[28731]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:26 smithi045 ceph-mon[36308]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:26 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:26 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:26 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:35:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:26 smithi044 ceph-mon[28731]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:35:29.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:28 smithi045 ceph-mon[36308]: pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:29.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:28 smithi044 ceph-mon[28731]: pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:31 smithi044 ceph-mon[28731]: pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:31 smithi044 ceph-mon[28731]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T11:35:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:31 smithi045 ceph-mon[36308]: pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:31 smithi045 ceph-mon[36308]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T11:35:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:31 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:31.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.804+0000 7fc703400700 1 -- 172.21.15.44:0/3230819598 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc70c104590 con 0x7fc6f0069210 2024-04-16T11:35:31.802 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 1 on host 'smithi044' 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fc6f0069210 msgr2=0x7fc6f006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fc6f0069210 0x7fc6f006b6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800b7a0 tx=0x7fc6f8009e30 comp rx=0 tx=0).stop 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 msgr2=0x7fc70c10fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 0x7fc70c10fb10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00c830 tx=0x7fc6fc004470 comp rx=0 tx=0).stop 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 shutdown_connections 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fc6f0069210 0x7fc6f006b6c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc70c0ff000 0x7fc70c10fb10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 --2- 172.21.15.44:0/3230819598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc70c1002b0 0x7fc70c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.807+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 >> 172.21.15.44:0/3230819598 conn(0x7fc70c0fa950 msgr2=0x7fc70c0fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:35:31.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.808+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 shutdown_connections 2024-04-16T11:35:31.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:31.808+0000 7fc712000700 1 -- 172.21.15.44:0/3230819598 wait complete. 2024-04-16T11:35:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:32 smithi044 ceph-mon[28731]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T11:35:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:32 smithi044 ceph-mon[28731]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T11:35:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:32 smithi044 ceph-mon[28731]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:35:32.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:32.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:32 smithi045 ceph-mon[36308]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T11:35:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:32 smithi045 ceph-mon[36308]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T11:35:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:32 smithi045 ceph-mon[36308]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:35:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:33.132 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:33 smithi044 ceph-mon[28731]: pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:33.132 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:33 smithi044 ceph-mon[28731]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-16T11:35:33.132 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:33 smithi044 ceph-mon[28731]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T11:35:33.132 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:33 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:33.132 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:33 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:33.137 DEBUG:teuthology.orchestra.run.smithi044:osd.1> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.1.service 2024-04-16T11:35:33.143 INFO:tasks.cephadm:Deploying osd.2 on smithi044 with /dev/vg_nvme/lv_2... 2024-04-16T11:35:33.143 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T11:35:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:33 smithi045 ceph-mon[36308]: pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:33 smithi045 ceph-mon[36308]: from='osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-16T11:35:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:33 smithi045 ceph-mon[36308]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T11:35:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:33 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:33 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:34.141 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:35:34.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:34 smithi044 ceph-mon[28731]: purged_snaps scrub starts 2024-04-16T11:35:34.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:34 smithi044 ceph-mon[28731]: purged_snaps scrub ok 2024-04-16T11:35:34.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:34 smithi044 ceph-mon[28731]: osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] boot 2024-04-16T11:35:34.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:34 smithi044 ceph-mon[28731]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T11:35:34.310 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:34 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:34 smithi045 ceph-mon[36308]: purged_snaps scrub starts 2024-04-16T11:35:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:34 smithi045 ceph-mon[36308]: purged_snaps scrub ok 2024-04-16T11:35:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:34 smithi045 ceph-mon[36308]: osd.1 [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] boot 2024-04-16T11:35:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:34 smithi045 ceph-mon[36308]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T11:35:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:34 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:35:35.314 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:35 smithi044 ceph-mon[28731]: pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:35.315 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:35 smithi044 ceph-mon[28731]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T11:35:35.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:35 smithi045 ceph-mon[36308]: pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T11:35:35.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:35 smithi045 ceph-mon[36308]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T11:35:36.942 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:36.942 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:36.942 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:36 smithi044 ceph-mon[28731]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:36 smithi045 ceph-mon[36308]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T11:35:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T11:35:37.571 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-16T11:35:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-16T11:35:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0353553 s, 297 MB/s 2024-04-16T11:35:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-16T11:35:38.882 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_2 2024-04-16T11:35:39.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:38 smithi044 ceph-mon[28731]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:38 smithi045 ceph-mon[36308]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:39.569 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:35:40.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:40 smithi044 ceph-mon[28731]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:40 smithi045 ceph-mon[36308]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:42.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 -- 172.21.15.44:0/4230944190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 msgr2=0x7fb35c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:35:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 --2- 172.21.15.44:0/4230944190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c0fd320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb34c004940 tx=0x7fb34c009e30 comp rx=0 tx=0).stop 2024-04-16T11:35:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 -- 172.21.15.44:0/4230944190 shutdown_connections 2024-04-16T11:35:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 --2- 172.21.15.44:0/4230944190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb35c0fe0e0 0x7fb35c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 --2- 172.21.15.44:0/4230944190 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c0fd320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 -- 172.21.15.44:0/4230944190 >> 172.21.15.44:0/4230944190 conn(0x7fb35c0f8530 msgr2=0x7fb35c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:35:42.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 -- 172.21.15.44:0/4230944190 shutdown_connections 2024-04-16T11:35:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.183+0000 7fb362600700 1 -- 172.21.15.44:0/4230944190 wait complete. 2024-04-16T11:35:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.184+0000 7fb362600700 1 Processor -- start 2024-04-16T11:35:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.184+0000 7fb362600700 1 -- start start 2024-04-16T11:35:42.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb362600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb362600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb35c0fe0e0 0x7fb35c110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb362600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c110870 con 0x7fb35c0fcef0 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb362600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c110fb0 con 0x7fb35c0fe0e0 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c10fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39096/0 (socket says 172.21.15.44:39096) 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35b400700 1 -- 172.21.15.44:0/3880266762 learned_addr learned my addr 172.21.15.44:0/3880266762 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35b400700 1 -- 172.21.15.44:0/3880266762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb35c0fe0e0 msgr2=0x7fb35c110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35aa00700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb35c0fe0e0 0x7fb35c110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35b400700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb35c0fe0e0 0x7fb35c110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.185+0000 7fb35b400700 1 -- 172.21.15.44:0/3880266762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c005040 con 0x7fb35c0fcef0 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.186+0000 7fb35b400700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c10fd30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00b3b0 tx=0x7fb34c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:35:42.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.186+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb34c00e040 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.186+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb34c00bcc0 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.186+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb34c01c630 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.186+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c111230 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.186+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c1117f0 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.187+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c101a70 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.188+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fb34c0139f0 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.189+0000 7fb353400700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb3400692f0 0x7fb34006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.189+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fb34c087250 con 0x7fb35c0fcef0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.189+0000 7fb35aa00700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb3400692f0 0x7fb34006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:35:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.190+0000 7fb35aa00700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb3400692f0 0x7fb34006b7a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb348004d60 tx=0x7fb34800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:35:42.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.194+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb34c057cc0 con 0x7fb35c0fcef0 2024-04-16T11:35:42.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:35:42.427+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb35c105b10 con 0x7fb3400692f0 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:35:43.210 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:35:43.211 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:43.211 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:35:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:35:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:43 smithi044 ceph-mon[28731]: Detected new or changed devices on smithi044 2024-04-16T11:35:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:43 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi044 to 6435M 2024-04-16T11:35:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:43 smithi044 ceph-mon[28731]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:35:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:43 smithi045 ceph-mon[36308]: Detected new or changed devices on smithi044 2024-04-16T11:35:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:43 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi044 to 6435M 2024-04-16T11:35:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:43 smithi045 ceph-mon[36308]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:35:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:44 smithi044 ceph-mon[28731]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:45.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:44 smithi045 ceph-mon[36308]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:46.209 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:45 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:35:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:46 smithi044 ceph-mon[28731]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:47.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:46 smithi045 ceph-mon[36308]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:47 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/398588649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0517235e-6140-460d-9e3c-378dddaea057"}]: dispatch 2024-04-16T11:35:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:47 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/398588649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0517235e-6140-460d-9e3c-378dddaea057"}]': finished 2024-04-16T11:35:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:47 smithi044 ceph-mon[28731]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T11:35:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:47 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:35:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:47 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/398588649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0517235e-6140-460d-9e3c-378dddaea057"}]: dispatch 2024-04-16T11:35:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:47 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/398588649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0517235e-6140-460d-9e3c-378dddaea057"}]': finished 2024-04-16T11:35:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:47 smithi045 ceph-mon[36308]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T11:35:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:47 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:35:49.326 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:49 smithi044 ceph-mon[28731]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:49.326 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:49 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2191558539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:35:49.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:49 smithi045 ceph-mon[36308]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:49.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:49 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2191558539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:35:51.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:51.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:51.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:51.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:51.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:51.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:50 smithi045 ceph-mon[36308]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:51.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:35:51.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:50 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:35:51.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:50 smithi044 ceph-mon[28731]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:53.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:52 smithi044 ceph-mon[28731]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:53.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:52 smithi045 ceph-mon[36308]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:55.128 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:54 smithi044 ceph-mon[28731]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:54 smithi045 ceph-mon[36308]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:56 smithi044 ceph-mon[28731]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T11:35:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:57.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:56 smithi045 ceph-mon[36308]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:57.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T11:35:57.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:35:58.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:57 smithi044 ceph-mon[28731]: Deploying daemon osd.2 on smithi044 2024-04-16T11:35:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:57 smithi045 ceph-mon[36308]: Deploying daemon osd.2 on smithi044 2024-04-16T11:35:58.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:35:58 smithi044 ceph-mon[28731]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:35:59.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:35:58 smithi045 ceph-mon[36308]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:01.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:00 smithi044 ceph-mon[28731]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:01.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:00 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:00 smithi045 ceph-mon[36308]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:00 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:02 smithi044 ceph-mon[28731]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:03.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:02 smithi045 ceph-mon[36308]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:05.051 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:04 smithi044 ceph-mon[28731]: pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:05.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:04 smithi045 ceph-mon[36308]: pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:06.299 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:06.300 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:06.300 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:07.301 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:07 smithi044 ceph-mon[28731]: pgmap v89: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:07 smithi045 ceph-mon[36308]: pgmap v89: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:09 smithi044 ceph-mon[28731]: pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:09.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:09 smithi044 ceph-mon[28731]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T11:36:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:09 smithi045 ceph-mon[36308]: pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:09 smithi045 ceph-mon[36308]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T11:36:10.202 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:10 smithi044 ceph-mon[28731]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T11:36:10.202 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:10 smithi044 ceph-mon[28731]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T11:36:10.203 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:10 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:10.203 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:10 smithi044 ceph-mon[28731]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:36:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:10 smithi045 ceph-mon[36308]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T11:36:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:10 smithi045 ceph-mon[36308]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T11:36:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:10 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:10 smithi045 ceph-mon[36308]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:36:10.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.780+0000 7fb353400700 1 -- 172.21.15.44:0/3880266762 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb35c105b10 con 0x7fb3400692f0 2024-04-16T11:36:10.780 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 2 on host 'smithi044' 2024-04-16T11:36:10.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb3400692f0 msgr2=0x7fb34006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:10.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb3400692f0 0x7fb34006b7a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb348004d60 tx=0x7fb34800b040 comp rx=0 tx=0).stop 2024-04-16T11:36:10.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 msgr2=0x7fb35c10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:10.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c10fd30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00b3b0 tx=0x7fb34c009e30 comp rx=0 tx=0).stop 2024-04-16T11:36:10.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 shutdown_connections 2024-04-16T11:36:10.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb3400692f0 0x7fb34006b7a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:10.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb35c0fcef0 0x7fb35c10fd30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:10.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 --2- 172.21.15.44:0/3880266762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb35c0fe0e0 0x7fb35c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:10.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.783+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 >> 172.21.15.44:0/3880266762 conn(0x7fb35c0f8530 msgr2=0x7fb35c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:36:10.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.784+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 shutdown_connections 2024-04-16T11:36:10.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:36:10.784+0000 7fb362600700 1 -- 172.21.15.44:0/3880266762 wait complete. 2024-04-16T11:36:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: pgmap v92: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-16T11:36:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T11:36:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:11.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: pgmap v92: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: from='osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:11.425 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:11.913 DEBUG:teuthology.orchestra.run.smithi044:osd.2> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2.service 2024-04-16T11:36:11.922 INFO:tasks.cephadm:Deploying osd.3 on smithi045 with /dev/vg_nvme/lv_4... 2024-04-16T11:36:11.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T11:36:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:36:12.791 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:12 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:12.792 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:12 smithi045 ceph-mon[36308]: osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] boot 2024-04-16T11:36:12.792 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:12 smithi045 ceph-mon[36308]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T11:36:12.792 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:12 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:12.798 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:12 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:12.798 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:12 smithi044 ceph-mon[28731]: osd.2 [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] boot 2024-04-16T11:36:12.798 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:12 smithi044 ceph-mon[28731]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T11:36:12.798 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:12 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:36:13.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: purged_snaps scrub starts 2024-04-16T11:36:13.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: purged_snaps scrub ok 2024-04-16T11:36:13.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: pgmap v95: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:13.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-16T11:36:13.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-16T11:36:13.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T11:36:13.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T11:36:13.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:13.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:13 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: purged_snaps scrub starts 2024-04-16T11:36:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: purged_snaps scrub ok 2024-04-16T11:36:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: pgmap v95: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T11:36:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-16T11:36:13.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-16T11:36:13.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T11:36:13.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T11:36:13.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:13.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:13 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:14.601 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T11:36:14.601 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:14.601 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:14.602 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64710]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:14.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:14 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T11:36:14.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:14 smithi045 ceph-mon[36308]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T11:36:14.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:14 smithi045 ceph-mon[36308]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:14.886 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:14 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T11:36:14.887 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:14 smithi044 ceph-mon[28731]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T11:36:14.887 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:14 smithi044 ceph-mon[28731]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T11:36:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T11:36:15.168 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-16T11:36:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-16T11:36:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0395852 s, 265 MB/s 2024-04-16T11:36:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-16T11:36:15.213 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64791]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T11:36:15.214 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64791]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:15.214 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64791]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:15.214 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:36:14 smithi044 sudo[64791]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:15.796 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:15 smithi045 ceph-mon[36308]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T11:36:15.796 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:15 smithi045 ceph-mon[36308]: mgrmap e19: smithi044.sczobv(active, since 2m), standbys: smithi045.sdlwif 2024-04-16T11:36:15.796 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:15 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:15.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-04-16T11:36:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:15 smithi044 ceph-mon[28731]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T11:36:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:15 smithi044 ceph-mon[28731]: mgrmap e19: smithi044.sczobv(active, since 2m), standbys: smithi045.sdlwif 2024-04-16T11:36:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:15.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:15 smithi044 sudo[64853]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T11:36:15.977 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:15 smithi044 sudo[64853]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:15.977 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:15 smithi044 sudo[64853]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:15.977 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:15 smithi044 sudo[64853]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:16.308 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:36:16.439 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64872]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T11:36:16.439 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64872]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:16.439 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64872]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:16.439 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64872]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:16.713 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:16 smithi044 ceph-mon[28731]: pgmap v100: 1 pgs: 1 active+clean; 449 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:16.713 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:16.749 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:16 smithi045 ceph-mon[36308]: pgmap v100: 1 pgs: 1 active+clean; 449 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:16.749 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:16.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64896]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T11:36:16.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64896]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:16.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64896]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:16.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:16 smithi044 sudo[64896]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:17.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T11:36:17.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:17.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64909]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:17.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64909]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64932]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T11:36:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64932]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64932]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:17 smithi044 sudo[64932]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.285+0000 7fe829e00700 1 -- 172.21.15.45:0/606660133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fe0e0 msgr2=0x7fe8240fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.285+0000 7fe829e00700 1 --2- 172.21.15.45:0/606660133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fe0e0 0x7fe8240fe550 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe80c004970 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.286+0000 7fe829e00700 1 -- 172.21.15.45:0/606660133 shutdown_connections 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.286+0000 7fe829e00700 1 --2- 172.21.15.45:0/606660133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fe0e0 0x7fe8240fe550 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.286+0000 7fe829e00700 1 --2- 172.21.15.45:0/606660133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8240fcef0 0x7fe8240fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.286+0000 7fe829e00700 1 -- 172.21.15.45:0/606660133 >> 172.21.15.45:0/606660133 conn(0x7fe8240f8530 msgr2=0x7fe8240fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.286+0000 7fe829e00700 1 -- 172.21.15.45:0/606660133 shutdown_connections 2024-04-16T11:36:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.286+0000 7fe829e00700 1 -- 172.21.15.45:0/606660133 wait complete. 2024-04-16T11:36:18.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.287+0000 7fe829e00700 1 Processor -- start 2024-04-16T11:36:18.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.287+0000 7fe829e00700 1 -- start start 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe829e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 0x7fe82410fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe829e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8240fe0e0 0x7fe824110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe829e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824110700 con 0x7fe8240fe0e0 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe829e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824110f50 con 0x7fe8240fcef0 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe822a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 0x7fe82410fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe822a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 0x7fe82410fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50576/0 (socket says 172.21.15.45:50576) 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe822a00700 1 -- 172.21.15.45:0/4180481008 learned_addr learned my addr 172.21.15.45:0/4180481008 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe81a000700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8240fe0e0 0x7fe824110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe822a00700 1 -- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8240fe0e0 msgr2=0x7fe824110100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe822a00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8240fe0e0 0x7fe824110100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:18.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.288+0000 7fe822a00700 1 -- 172.21.15.45:0/4180481008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c005040 con 0x7fe8240fcef0 2024-04-16T11:36:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.289+0000 7fe822a00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 0x7fe82410fbc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe81400ca90 tx=0x7fe814007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:36:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.289+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe81400aa50 con 0x7fe8240fcef0 2024-04-16T11:36:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.289+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe81400abb0 con 0x7fe8240fcef0 2024-04-16T11:36:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.289+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8140115e0 con 0x7fe8240fcef0 2024-04-16T11:36:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.289+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe824111230 con 0x7fe8240fcef0 2024-04-16T11:36:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.289+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe824111770 con 0x7fe8240fcef0 2024-04-16T11:36:18.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.290+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe824101a70 con 0x7fe8240fcef0 2024-04-16T11:36:18.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.292+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe81400ad20 con 0x7fe8240fcef0 2024-04-16T11:36:18.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.293+0000 7fe81b400700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe808069360 0x7fe80806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:36:18.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.293+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7fe814086f00 con 0x7fe8240fcef0 2024-04-16T11:36:18.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.293+0000 7fe81a000700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe808069360 0x7fe80806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:36:18.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.294+0000 7fe81a000700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe808069360 0x7fe80806b810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00b800 tx=0x7fe80c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:36:18.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.298+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe814057560 con 0x7fe8240fcef0 2024-04-16T11:36:18.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:18.522+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8241119e0 con 0x7fe808069360 2024-04-16T11:36:18.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:18 smithi044 sudo[66491]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T11:36:18.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:18 smithi044 sudo[66491]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:18.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:18 smithi044 sudo[66491]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:18.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:18 smithi044 sudo[66491]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:19.213 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T11:36:19.213 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:19.213 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39574]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:19.213 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39574]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:19.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39610]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T11:36:19.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39610]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T11:36:19.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39610]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T11:36:19.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 sudo[39610]: pam_unix(sudo:session): session closed for user root 2024-04-16T11:36:19.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: pgmap v101: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:36:19.835 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:19.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T11:36:19.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: pgmap v101: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:19.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T11:36:19.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:36:19.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:36:19.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:36:19.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:36:19.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:36:19.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:36:19.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:20.763 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T11:36:20.763 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:36:20.763 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: Detected new or changed devices on smithi044 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi044 to 4290M 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 99 KiB/s wr, 6 op/s 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:20.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:20.765 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:20.765 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:20 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: Detected new or changed devices on smithi044 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi044 to 4290M 2024-04-16T11:36:20.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 99 KiB/s wr, 6 op/s 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:20.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:20 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2024-04-16T11:36:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2531193525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37db698-8398-425d-ac76-25d6f4c38505"}]: dispatch 2024-04-16T11:36:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37db698-8398-425d-ac76-25d6f4c38505"}]: dispatch 2024-04-16T11:36:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d37db698-8398-425d-ac76-25d6f4c38505"}]': finished 2024-04-16T11:36:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: osdmap e23: 4 total, 3 up, 4 in 2024-04-16T11:36:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:23.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:22 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/3214188067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/2531193525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37db698-8398-425d-ac76-25d6f4c38505"}]: dispatch 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37db698-8398-425d-ac76-25d6f4c38505"}]: dispatch 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d37db698-8398-425d-ac76-25d6f4c38505"}]': finished 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: osdmap e23: 4 total, 3 up, 4 in 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:22 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/3214188067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:36:25.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:24 smithi044 ceph-mon[28731]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 992 B/s rd, 56 KiB/s wr, 3 op/s 2024-04-16T11:36:25.303 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:24 smithi045 ceph-mon[36308]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 992 B/s rd, 56 KiB/s wr, 3 op/s 2024-04-16T11:36:27.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:26 smithi044 ceph-mon[28731]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T11:36:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:26 smithi045 ceph-mon[36308]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T11:36:29.051 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:28 smithi045 ceph-mon[36308]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T11:36:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:28 smithi044 ceph-mon[28731]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-16T11:36:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:30 smithi044 ceph-mon[28731]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T11:36:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T11:36:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:31.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:30 smithi045 ceph-mon[36308]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T11:36:31.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:30 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T11:36:31.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:30 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:31.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:30 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:32.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:31 smithi045 ceph-mon[36308]: Deploying daemon osd.3 on smithi045 2024-04-16T11:36:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:31 smithi044 ceph-mon[28731]: Deploying daemon osd.3 on smithi045 2024-04-16T11:36:33.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:32 smithi044 ceph-mon[28731]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T11:36:33.253 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:32 smithi045 ceph-mon[36308]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T11:36:35.299 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:34 smithi045 ceph-mon[36308]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 66 KiB/s wr, 4 op/s 2024-04-16T11:36:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:34 smithi044 ceph-mon[28731]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 66 KiB/s wr, 4 op/s 2024-04-16T11:36:37.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:36 smithi045 ceph-mon[36308]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T11:36:37.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:36 smithi044 ceph-mon[28731]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T11:36:38.986 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:38 smithi045 ceph-mon[36308]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:38.986 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:38.986 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:38.986 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:38 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:39.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:38 smithi044 ceph-mon[28731]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:39.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:39.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:39.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:38 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:40 smithi045 ceph-mon[36308]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:40 smithi044 ceph-mon[28731]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:42.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.380+0000 7fe81b400700 1 -- 172.21.15.45:0/4180481008 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe8241119e0 con 0x7fe808069360 2024-04-16T11:36:42.380 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 3 on host 'smithi045' 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe808069360 msgr2=0x7fe80806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe808069360 0x7fe80806b810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00b800 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 msgr2=0x7fe82410fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 0x7fe82410fbc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe81400ca90 tx=0x7fe814007f00 comp rx=0 tx=0).stop 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 shutdown_connections 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe808069360 0x7fe80806b810 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8240fcef0 0x7fe82410fbc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 --2- 172.21.15.45:0/4180481008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8240fe0e0 0x7fe824110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 >> 172.21.15.45:0/4180481008 conn(0x7fe8240f8530 msgr2=0x7fe8240f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:36:42.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 shutdown_connections 2024-04-16T11:36:42.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:42.384+0000 7fe829e00700 1 -- 172.21.15.45:0/4180481008 wait complete. 2024-04-16T11:36:43.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:43.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: from='osd.3 [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:36:43.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:36:43.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: from='osd.3 [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:36:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:36:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:43.356 DEBUG:teuthology.orchestra.run.smithi045:osd.3> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3.service 2024-04-16T11:36:43.362 INFO:tasks.cephadm:Deploying osd.4 on smithi045 with /dev/vg_nvme/lv_3... 2024-04-16T11:36:43.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T11:36:44.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-mon[36308]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T11:36:44.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-mon[36308]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T11:36:44.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:44.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-mon[36308]: from='osd.3 [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:36:44.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-mon[36308]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:36:44.220 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[41815]: 2024-04-16T11:36:43.963+0000 7f3c42e00700 -1 osd.3 0 waiting for initial osdmap 2024-04-16T11:36:44.220 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:36:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[41815]: 2024-04-16T11:36:43.980+0000 7f3c3d400700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:36:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:43 smithi044 ceph-mon[28731]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T11:36:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:43 smithi044 ceph-mon[28731]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T11:36:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:43 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:43 smithi044 ceph-mon[28731]: from='osd.3 [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:36:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:43 smithi044 ceph-mon[28731]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:36:44.556 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:36:45.119 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:44 smithi045 ceph-mon[36308]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:45.119 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:44 smithi045 ceph-mon[36308]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-16T11:36:45.119 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:44 smithi045 ceph-mon[36308]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T11:36:45.119 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:45.119 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:44 smithi044 ceph-mon[28731]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:44 smithi044 ceph-mon[28731]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-16T11:36:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:44 smithi044 ceph-mon[28731]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T11:36:45.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:45.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: purged_snaps scrub starts 2024-04-16T11:36:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: purged_snaps scrub ok 2024-04-16T11:36:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: osd.3 [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] boot 2024-04-16T11:36:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: osdmap e26: 4 total, 4 up, 4 in 2024-04-16T11:36:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T11:36:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: purged_snaps scrub starts 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: purged_snaps scrub ok 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: osd.3 [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] boot 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: osdmap e26: 4 total, 4 up, 4 in 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T11:36:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:45 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:36:47.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:46 smithi045 ceph-mon[36308]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:47.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:46 smithi045 ceph-mon[36308]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T11:36:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:46 smithi044 ceph-mon[28731]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T11:36:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:46 smithi044 ceph-mon[28731]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T11:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T11:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T11:36:47.636 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-16T11:36:47.637 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-16T11:36:47.637 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0360783 s, 291 MB/s 2024-04-16T11:36:47.637 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-16T11:36:48.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-04-16T11:36:48.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T11:36:48.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: Detected new or changed devices on smithi045 2024-04-16T11:36:48.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi045 to 15942M 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:48.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: Detected new or changed devices on smithi045 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi045 to 15942M 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:36:48.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:48.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:36:48.728 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:36:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:36:49.463 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:49 smithi045 ceph-mon[36308]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T11:36:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:49 smithi044 ceph-mon[28731]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T11:36:50.719 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:50 smithi045 ceph-mon[36308]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-16T11:36:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:50 smithi044 ceph-mon[28731]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-16T11:36:50.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 -- 172.21.15.45:0/277450091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc1046b0 msgr2=0x7fa2dc104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 --2- 172.21.15.45:0/277450091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc104b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8004940 tx=0x7fa2c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 -- 172.21.15.45:0/277450091 shutdown_connections 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 --2- 172.21.15.45:0/277450091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc104b20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 --2- 172.21.15.45:0/277450091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc103400 0x7fa2dc103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 -- 172.21.15.45:0/277450091 >> 172.21.15.45:0/277450091 conn(0x7fa2dc0725e0 msgr2=0x7fa2dc074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 -- 172.21.15.45:0/277450091 shutdown_connections 2024-04-16T11:36:50.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.889+0000 7fa2e3000700 1 -- 172.21.15.45:0/277450091 wait complete. 2024-04-16T11:36:50.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2e3000700 1 Processor -- start 2024-04-16T11:36:50.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2e3000700 1 -- start start 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2e3000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 0x7fa2dc113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2e3000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2e3000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc114a50 con 0x7fa2dc1046b0 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2e3000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc114bc0 con 0x7fa2dc103400 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.891+0000 7fa2dbe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 0x7fa2dc113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2dbe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 0x7fa2dc113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35386/0 (socket says 172.21.15.45:35386) 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2dbe00700 1 -- 172.21.15.45:0/3918345437 learned_addr learned my addr 172.21.15.45:0/3918345437 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2db400700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2dbe00700 1 -- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc1046b0 msgr2=0x7fa2dc114450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2dbe00700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc114450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2dbe00700 1 -- 172.21.15.45:0/3918345437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8005040 con 0x7fa2dc103400 2024-04-16T11:36:50.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2db400700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc114450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:36:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.892+0000 7fa2dbe00700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 0x7fa2dc113f10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc00c8c0 tx=0x7fa2cc0044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:36:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.893+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2cc00a980 con 0x7fa2dc103400 2024-04-16T11:36:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.893+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2cc00ce60 con 0x7fa2dc103400 2024-04-16T11:36:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.893+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2cc0115e0 con 0x7fa2dc103400 2024-04-16T11:36:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.893+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc119620 con 0x7fa2dc103400 2024-04-16T11:36:50.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.893+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc119be0 con 0x7fa2dc103400 2024-04-16T11:36:50.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.894+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc04c710 con 0x7fa2dc103400 2024-04-16T11:36:50.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.895+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa2cc012400 con 0x7fa2dc103400 2024-04-16T11:36:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.896+0000 7fa2d3e00700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2c00692a0 0x7fa2c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:36:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.896+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fa2cc086ca0 con 0x7fa2dc103400 2024-04-16T11:36:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.897+0000 7fa2db400700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2c00692a0 0x7fa2c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:36:50.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.898+0000 7fa2db400700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2c00692a0 0x7fa2c006b750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800b020 tx=0x7fa2c8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:36:50.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:50.902+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa2cc057060 con 0x7fa2dc103400 2024-04-16T11:36:51.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:36:51.129+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2dc1089d0 con 0x7fa2c00692a0 2024-04-16T11:36:51.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:51 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:36:51.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:51 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:36:51.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:51 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:51 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:36:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:51 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:36:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:51 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:36:52.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:52 smithi045 ceph-mon[36308]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:36:52.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:52 smithi045 ceph-mon[36308]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-16T11:36:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:52 smithi044 ceph-mon[28731]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:36:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:52 smithi044 ceph-mon[28731]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-16T11:36:55.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:54 smithi044 ceph-mon[28731]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-16T11:36:55.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:54 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2300393028' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d58def5c-f1a7-42d0-8e0f-95acdc12b5e9"}]: dispatch 2024-04-16T11:36:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:54 smithi044 ceph-mon[28731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d58def5c-f1a7-42d0-8e0f-95acdc12b5e9"}]: dispatch 2024-04-16T11:36:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:54 smithi044 ceph-mon[28731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d58def5c-f1a7-42d0-8e0f-95acdc12b5e9"}]': finished 2024-04-16T11:36:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:54 smithi044 ceph-mon[28731]: osdmap e31: 5 total, 4 up, 5 in 2024-04-16T11:36:55.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:36:55.319 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:54 smithi045 ceph-mon[36308]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-16T11:36:55.320 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:54 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/2300393028' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d58def5c-f1a7-42d0-8e0f-95acdc12b5e9"}]: dispatch 2024-04-16T11:36:55.320 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:54 smithi045 ceph-mon[36308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d58def5c-f1a7-42d0-8e0f-95acdc12b5e9"}]: dispatch 2024-04-16T11:36:55.320 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:54 smithi045 ceph-mon[36308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d58def5c-f1a7-42d0-8e0f-95acdc12b5e9"}]': finished 2024-04-16T11:36:55.320 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:54 smithi045 ceph-mon[36308]: osdmap e31: 5 total, 4 up, 5 in 2024-04-16T11:36:55.320 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:36:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:55 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/2555279688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:36:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:55 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/2555279688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:36:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:56 smithi044 ceph-mon[28731]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-16T11:36:57.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:56 smithi045 ceph-mon[36308]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-16T11:36:59.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:36:58 smithi045 ceph-mon[36308]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-16T11:36:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:36:58 smithi044 ceph-mon[28731]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-16T11:37:01.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:00 smithi044 ceph-mon[28731]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:01.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:00 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:01.275 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:00 smithi045 ceph-mon[36308]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:01.276 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:00 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:02 smithi044 ceph-mon[28731]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:03.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:02 smithi045 ceph-mon[36308]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:04.205 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:03 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T11:37:04.206 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:03 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:04.206 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:03 smithi045 ceph-mon[36308]: Deploying daemon osd.4 on smithi045 2024-04-16T11:37:04.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:03 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T11:37:04.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:03 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:04.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:03 smithi044 ceph-mon[28731]: Deploying daemon osd.4 on smithi045 2024-04-16T11:37:05.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:04 smithi045 ceph-mon[36308]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:04 smithi044 ceph-mon[28731]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:07.288 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:06 smithi045 ceph-mon[36308]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:06 smithi044 ceph-mon[28731]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:09.136 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:09 smithi045 ceph-mon[36308]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:09.279 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:09 smithi044 ceph-mon[28731]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:11 smithi045 ceph-mon[36308]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:11 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:11 smithi044 ceph-mon[28731]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:11 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:13.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:13 smithi045 ceph-mon[36308]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:13.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:13 smithi044 ceph-mon[28731]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:14.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.936+0000 7fa2d3e00700 1 -- 172.21.15.45:0/3918345437 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa2dc1089d0 con 0x7fa2c00692a0 2024-04-16T11:37:14.935 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 4 on host 'smithi045' 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2c00692a0 msgr2=0x7fa2c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2c00692a0 0x7fa2c006b750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800b020 tx=0x7fa2c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 msgr2=0x7fa2dc113f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 0x7fa2dc113f10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc00c8c0 tx=0x7fa2cc0044c0 comp rx=0 tx=0).stop 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 shutdown_connections 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2c00692a0 0x7fa2c006b750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2dc103400 0x7fa2dc113f10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 --2- 172.21.15.45:0/3918345437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2dc1046b0 0x7fa2dc114450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 >> 172.21.15.45:0/3918345437 conn(0x7fa2dc0725e0 msgr2=0x7fa2dc073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 shutdown_connections 2024-04-16T11:37:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:14.939+0000 7fa2e3000700 1 -- 172.21.15.45:0/3918345437 wait complete. 2024-04-16T11:37:15.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:15 smithi045 ceph-mon[36308]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:15 smithi045 ceph-mon[36308]: from='osd.4 [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:37:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:15 smithi045 ceph-mon[36308]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:37:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:15 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:15 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:15.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:15 smithi044 ceph-mon[28731]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:15.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:15 smithi044 ceph-mon[28731]: from='osd.4 [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:37:15.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:15 smithi044 ceph-mon[28731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:37:15.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:15.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:15 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:15.713 DEBUG:teuthology.orchestra.run.smithi045:osd.4> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.4.service 2024-04-16T11:37:15.719 INFO:tasks.cephadm:Deploying osd.5 on smithi045 with /dev/vg_nvme/lv_2... 2024-04-16T11:37:15.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T11:37:16.226 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:37:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='osd.4 [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:16.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='osd.4 [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:16.331 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[48372]: 2024-04-16T11:37:16.054+0000 7ffbba200700 -1 osd.4 0 waiting for initial osdmap 2024-04-16T11:37:16.331 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:37:16 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[48372]: 2024-04-16T11:37:16.069+0000 7ffbb1600700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:37:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:17 smithi045 ceph-mon[36308]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:17 smithi045 ceph-mon[36308]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-16T11:37:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:17 smithi045 ceph-mon[36308]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T11:37:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:17 smithi044 ceph-mon[28731]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:17 smithi044 ceph-mon[28731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-16T11:37:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:17 smithi044 ceph-mon[28731]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T11:37:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:18.250 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:18 smithi045 ceph-mon[36308]: purged_snaps scrub starts 2024-04-16T11:37:18.251 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:18 smithi045 ceph-mon[36308]: purged_snaps scrub ok 2024-04-16T11:37:18.251 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:18 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:18.251 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:18 smithi045 ceph-mon[36308]: osd.4 [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] boot 2024-04-16T11:37:18.251 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:18 smithi045 ceph-mon[36308]: osdmap e34: 5 total, 5 up, 5 in 2024-04-16T11:37:18.251 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:18 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:18 smithi044 ceph-mon[28731]: purged_snaps scrub starts 2024-04-16T11:37:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:18 smithi044 ceph-mon[28731]: purged_snaps scrub ok 2024-04-16T11:37:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:18 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:18 smithi044 ceph-mon[28731]: osd.4 [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] boot 2024-04-16T11:37:18.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:18 smithi044 ceph-mon[28731]: osdmap e34: 5 total, 5 up, 5 in 2024-04-16T11:37:18.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:18 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:37:19.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:19 smithi045 ceph-mon[36308]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:19.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:19 smithi045 ceph-mon[36308]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T11:37:19.160 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T11:37:19.160 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T11:37:19.160 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-16T11:37:19.160 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-16T11:37:19.160 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0368233 s, 285 MB/s 2024-04-16T11:37:19.160 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-16T11:37:19.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:19 smithi044 ceph-mon[28731]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T11:37:19.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:19 smithi044 ceph-mon[28731]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T11:37:19.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-04-16T11:37:20.456 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:37:21.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:21 smithi045 ceph-mon[36308]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:21 smithi044 ceph-mon[28731]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:23.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.840+0000 7f698be00700 1 -- 172.21.15.45:0/3365087392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fcff0 msgr2=0x7f69840fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 --2- 172.21.15.45:0/3365087392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fcff0 0x7f69840fd460 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6970004940 tx=0x7f6970009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 -- 172.21.15.45:0/3365087392 shutdown_connections 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 --2- 172.21.15.45:0/3365087392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fcff0 0x7f69840fd460 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 --2- 172.21.15.45:0/3365087392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fbd40 0x7f69840fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 -- 172.21.15.45:0/3365087392 >> 172.21.15.45:0/3365087392 conn(0x7f69840f74c0 msgr2=0x7f69840f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 -- 172.21.15.45:0/3365087392 shutdown_connections 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.841+0000 7f698be00700 1 -- 172.21.15.45:0/3365087392 wait complete. 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.842+0000 7f698be00700 1 Processor -- start 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.842+0000 7f698be00700 1 -- start start 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f698be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 0x7f698410fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:23.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f698be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fcff0 0x7f6984110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f698be00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6984110770 con 0x7f69840fcff0 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f698be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6984110fc0 con 0x7f69840fbd40 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f6989400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 0x7f698410fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f6989400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 0x7f698410fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49978/0 (socket says 172.21.15.45:49978) 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f6989400700 1 -- 172.21.15.45:0/2054967753 learned_addr learned my addr 172.21.15.45:0/2054967753 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.843+0000 7f6988a00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fcff0 0x7f6984110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6989400700 1 -- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fcff0 msgr2=0x7f6984110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6989400700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fcff0 0x7f6984110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6989400700 1 -- 172.21.15.45:0/2054967753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6970005040 con 0x7f69840fbd40 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6988a00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fcff0 0x7f6984110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6989400700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 0x7f698410fc30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6978002a50 tx=0x7f69780046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6978007cc0 con 0x7f69840fbd40 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f697800f040 con 0x7f69840fbd40 2024-04-16T11:37:23.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69841112a0 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6978009640 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.844+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69841117e0 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.845+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6984101980 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.846+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f69780097a0 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.847+0000 7f6981600700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f696c0692b0 0x7f696c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.847+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f6978086b80 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.848+0000 7f6988a00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f696c0692b0 0x7f696c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.849+0000 7f6988a00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f696c0692b0 0x7f696c06b760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f697000b7a0 tx=0x7f6970009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:22.852+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6978056d10 con 0x7f69840fbd40 2024-04-16T11:37:23.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:23.082+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6984111a00 con 0x7f696c0692b0 2024-04-16T11:37:23.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:23 smithi045 ceph-mon[36308]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:23 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:23 smithi044 ceph-mon[28731]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:23 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: Detected new or changed devices on smithi045 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi045 to 7971M 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:24.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:24 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: Detected new or changed devices on smithi045 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi045 to 7971M 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:24 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:25.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:25 smithi045 ceph-mon[36308]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:25.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:25 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:25.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:25 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:25.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:25 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:25 smithi044 ceph-mon[28731]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:25 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:27 smithi044 ceph-mon[28731]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:27 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/1658900556' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cc55233-3bbf-40ec-b61d-71fe16d1c35d"}]: dispatch 2024-04-16T11:37:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:27 smithi044 ceph-mon[28731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cc55233-3bbf-40ec-b61d-71fe16d1c35d"}]: dispatch 2024-04-16T11:37:27.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:27 smithi044 ceph-mon[28731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cc55233-3bbf-40ec-b61d-71fe16d1c35d"}]': finished 2024-04-16T11:37:27.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:27 smithi044 ceph-mon[28731]: osdmap e36: 6 total, 5 up, 6 in 2024-04-16T11:37:27.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:27 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:27 smithi045 ceph-mon[36308]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:27 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/1658900556' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cc55233-3bbf-40ec-b61d-71fe16d1c35d"}]: dispatch 2024-04-16T11:37:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:27 smithi045 ceph-mon[36308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cc55233-3bbf-40ec-b61d-71fe16d1c35d"}]: dispatch 2024-04-16T11:37:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:27 smithi045 ceph-mon[36308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cc55233-3bbf-40ec-b61d-71fe16d1c35d"}]': finished 2024-04-16T11:37:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:27 smithi045 ceph-mon[36308]: osdmap e36: 6 total, 5 up, 6 in 2024-04-16T11:37:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:27 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:28.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:28 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/4118910879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:37:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:28 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/4118910879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T11:37:29.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:29 smithi044 ceph-mon[28731]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:29 smithi045 ceph-mon[36308]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:31.270 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:31 smithi045 ceph-mon[36308]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:31.270 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:31 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:31 smithi044 ceph-mon[28731]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:33.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:33 smithi044 ceph-mon[28731]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:33.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:33 smithi045 ceph-mon[36308]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:35.370 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:35 smithi045 ceph-mon[36308]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:35 smithi044 ceph-mon[28731]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:36.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T11:37:36.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:36.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T11:37:36.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:37.243 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:37 smithi045 ceph-mon[36308]: Deploying daemon osd.5 on smithi045 2024-04-16T11:37:37.243 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:37 smithi045 ceph-mon[36308]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:37.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:37 smithi044 ceph-mon[28731]: Deploying daemon osd.5 on smithi045 2024-04-16T11:37:37.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:37 smithi044 ceph-mon[28731]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:38.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:38 smithi045 ceph-mon[36308]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:38.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:38 smithi044 ceph-mon[28731]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:40 smithi045 ceph-mon[36308]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:40 smithi044 ceph-mon[28731]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:43.129 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:42 smithi045 ceph-mon[36308]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:43.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:42 smithi044 ceph-mon[28731]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:44.389 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:44.389 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:44.389 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:44.389 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:44 smithi045 ceph-mon[36308]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:44 smithi044 ceph-mon[28731]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:46.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:45 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:37:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:46 smithi044 ceph-mon[28731]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:47.232 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:46 smithi045 ceph-mon[36308]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:47.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.677+0000 7f6981600700 1 -- 172.21.15.45:0/2054967753 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6984111a00 con 0x7f696c0692b0 2024-04-16T11:37:47.677 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 5 on host 'smithi045' 2024-04-16T11:37:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f696c0692b0 msgr2=0x7f696c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f696c0692b0 0x7f696c06b760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f697000b7a0 tx=0x7f6970009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 msgr2=0x7f698410fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 0x7f698410fc30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6978002a50 tx=0x7f69780046b0 comp rx=0 tx=0).stop 2024-04-16T11:37:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 shutdown_connections 2024-04-16T11:37:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f696c0692b0 0x7f696c06b760 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69840fbd40 0x7f698410fc30 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6978002a50 tx=0x7f69780046b0 comp rx=0 tx=0).stop 2024-04-16T11:37:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 --2- 172.21.15.45:0/2054967753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69840fcff0 0x7f6984110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.680+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 >> 172.21.15.45:0/2054967753 conn(0x7f69840f74c0 msgr2=0x7f6984065b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.681+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 shutdown_connections 2024-04-16T11:37:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:37:47.681+0000 7f698be00700 1 -- 172.21.15.45:0/2054967753 wait complete. 2024-04-16T11:37:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:47 smithi044 ceph-mon[28731]: from='osd.5 [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:37:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:47 smithi044 ceph-mon[28731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:37:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:47 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:47 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:47 smithi045 ceph-mon[36308]: from='osd.5 [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:37:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:47 smithi045 ceph-mon[36308]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:37:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:47 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:47 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:48.425 DEBUG:teuthology.orchestra.run.smithi045:osd.5> sudo journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.5.service 2024-04-16T11:37:48.431 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-16T11:37:48.431 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-16T11:37:48.955 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:37:49.262 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: from='osd.5 [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:49.263 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:49.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: from='osd.5 [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:49.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:49.306 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:37:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[55147]: 2024-04-16T11:37:48.998+0000 7fec6d200700 -1 osd.5 0 waiting for initial osdmap 2024-04-16T11:37:49.306 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:37:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[55147]: 2024-04-16T11:37:49.009+0000 7fec66400700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:37:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:49 smithi044 ceph-mon[28731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-16T11:37:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:49 smithi044 ceph-mon[28731]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T11:37:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:49 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:49 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:50 smithi045 ceph-mon[36308]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-16T11:37:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:50 smithi045 ceph-mon[36308]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T11:37:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:50 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:51.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.294+0000 7fa319400700 1 -- 172.21.15.44:0/4172581684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 msgr2=0x7fa3140ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:51.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.294+0000 7fa319400700 1 --2- 172.21.15.44:0/4172581684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa3140ff260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa304004940 tx=0x7fa304009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.294+0000 7fa319400700 1 -- 172.21.15.44:0/4172581684 shutdown_connections 2024-04-16T11:37:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.294+0000 7fa319400700 1 --2- 172.21.15.44:0/4172581684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 0x7fa314100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.294+0000 7fa319400700 1 --2- 172.21.15.44:0/4172581684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa3140ff260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.294+0000 7fa319400700 1 -- 172.21.15.44:0/4172581684 >> 172.21.15.44:0/4172581684 conn(0x7fa3140fa5d0 msgr2=0x7fa3140fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.295+0000 7fa319400700 1 -- 172.21.15.44:0/4172581684 shutdown_connections 2024-04-16T11:37:51.293 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: purged_snaps scrub starts 2024-04-16T11:37:51.293 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: purged_snaps scrub ok 2024-04-16T11:37:51.293 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:51.294 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:51.294 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: osd.5 [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] boot 2024-04-16T11:37:51.294 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: osdmap e39: 6 total, 6 up, 6 in 2024-04-16T11:37:51.294 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:51 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.295+0000 7fa319400700 1 -- 172.21.15.44:0/4172581684 wait complete. 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.296+0000 7fa319400700 1 Processor -- start 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.296+0000 7fa319400700 1 -- start start 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa319400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa31410fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa319400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 0x7fa31410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa319400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3141105a0 con 0x7fa3140fee50 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa319400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa314110df0 con 0x7fa3140fffc0 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa313e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa31410fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa313e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa31410fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51800/0 (socket says 172.21.15.44:51800) 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa313e00700 1 -- 172.21.15.44:0/3129094896 learned_addr learned my addr 172.21.15.44:0/3129094896 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.297+0000 7fa30be00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 0x7fa31410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa313e00700 1 -- 172.21.15.44:0/3129094896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 msgr2=0x7fa31410ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa313e00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 0x7fa31410ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa313e00700 1 -- 172.21.15.44:0/3129094896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304005040 con 0x7fa3140fee50 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa30be00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 0x7fa31410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa313e00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa31410fa60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa30400b3b0 tx=0x7fa304009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30400e040 con 0x7fa3140fee50 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa30400bcc0 con 0x7fa3140fee50 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa314111070 con 0x7fa3140fee50 2024-04-16T11:37:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.298+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30401c630 con 0x7fa3140fee50 2024-04-16T11:37:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.299+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3141115b0 con 0x7fa3140fee50 2024-04-16T11:37:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.300+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31405f5f0 con 0x7fa3140fee50 2024-04-16T11:37:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.301+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa304013980 con 0x7fa3140fee50 2024-04-16T11:37:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.302+0000 7fa310c00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2f80692a0 0x7fa2f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.302+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa304087c50 con 0x7fa3140fee50 2024-04-16T11:37:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.302+0000 7fa30be00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2f80692a0 0x7fa2f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:51.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.303+0000 7fa30be00700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2f80692a0 0x7fa2f806b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009fd0 tx=0x7fa2fc009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:51.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.307+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa304057cb0 con 0x7fa3140fee50 2024-04-16T11:37:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: purged_snaps scrub starts 2024-04-16T11:37:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: purged_snaps scrub ok 2024-04-16T11:37:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T11:37:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: osd.5 [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] boot 2024-04-16T11:37:51.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: osdmap e39: 6 total, 6 up, 6 in 2024-04-16T11:37:51.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:51 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:37:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.524+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa31404bed0 con 0x7fa3140fee50 2024-04-16T11:37:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.525+0000 7fa310c00700 1 -- 172.21.15.44:0/3129094896 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa304057840 con 0x7fa3140fee50 2024-04-16T11:37:51.522 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:37:51.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.527+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2f80692a0 msgr2=0x7fa2f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:51.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.527+0000 7fa319400700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2f80692a0 0x7fa2f806b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009fd0 tx=0x7fa2fc009ee0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.527+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 msgr2=0x7fa31410fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.527+0000 7fa319400700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa31410fa60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa30400b3b0 tx=0x7fa304009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 shutdown_connections 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2f80692a0 0x7fa2f806b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3140fee50 0x7fa31410fa60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 --2- 172.21.15.44:0/3129094896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3140fffc0 0x7fa31410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 >> 172.21.15.44:0/3129094896 conn(0x7fa3140fa5d0 msgr2=0x7fa3140fbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 shutdown_connections 2024-04-16T11:37:51.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:51.528+0000 7fa319400700 1 -- 172.21.15.44:0/3129094896 wait complete. 2024-04-16T11:37:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:52 smithi044 ceph-mon[28731]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T11:37:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:52 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3129094896' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T11:37:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:52 smithi045 ceph-mon[36308]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T11:37:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:52 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3129094896' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T11:37:52.367 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1713267469,"num_in_osds":6,"osd_in_since":1713267446,"num_remapped_pgs":0} 2024-04-16T11:37:52.368 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T11:37:52.825 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:37:53.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:53 smithi044 ceph-mon[28731]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:53.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:53 smithi045 ceph-mon[36308]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: Detected new or changed devices on smithi045 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi045 to 5314M 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:54.910 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:54.911 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:54 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:54.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 -- 172.21.15.44:0/1447151624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec600656b0 msgr2=0x7fec60065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 --2- 172.21.15.44:0/1447151624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec600656b0 0x7fec60065b20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fec4c004940 tx=0x7fec4c009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 -- 172.21.15.44:0/1447151624 shutdown_connections 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 --2- 172.21.15.44:0/1447151624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec600656b0 0x7fec60065b20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 --2- 172.21.15.44:0/1447151624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec60102e60 0x7fec60103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 -- 172.21.15.44:0/1447151624 >> 172.21.15.44:0/1447151624 conn(0x7fec60072bc0 msgr2=0x7fec60072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 -- 172.21.15.44:0/1447151624 shutdown_connections 2024-04-16T11:37:54.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.984+0000 7fec65a00700 1 -- 172.21.15.44:0/1447151624 wait complete. 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.985+0000 7fec65a00700 1 Processor -- start 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.985+0000 7fec65a00700 1 -- start start 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.985+0000 7fec65a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec600656b0 0x7fec6010b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.985+0000 7fec65a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 0x7fec6010bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.985+0000 7fec65a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6010c380 con 0x7fec60102e60 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.985+0000 7fec65a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6010c4c0 con 0x7fec600656b0 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 0x7fec6010bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 0x7fec6010bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50972/0 (socket says 172.21.15.44:50972) 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5e000700 1 -- 172.21.15.44:0/3878198779 learned_addr learned my addr 172.21.15.44:0/3878198779 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5ea00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec600656b0 0x7fec6010b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5e000700 1 -- 172.21.15.44:0/3878198779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec600656b0 msgr2=0x7fec6010b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5e000700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec600656b0 0x7fec6010b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:54.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.986+0000 7fec5e000700 1 -- 172.21.15.44:0/3878198779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c005040 con 0x7fec60102e60 2024-04-16T11:37:54.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.987+0000 7fec5e000700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 0x7fec6010bdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00b8b0 tx=0x7fec4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:54.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.987+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec4c00e040 con 0x7fec60102e60 2024-04-16T11:37:54.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.987+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec4c004290 con 0x7fec60102e60 2024-04-16T11:37:54.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.987+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec4c01c3f0 con 0x7fec60102e60 2024-04-16T11:37:54.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.987+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6011e250 con 0x7fec60102e60 2024-04-16T11:37:54.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.987+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6011e6d0 con 0x7fec60102e60 2024-04-16T11:37:54.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.988+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec60107320 con 0x7fec60102e60 2024-04-16T11:37:54.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.989+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fec4c0137b0 con 0x7fec60102e60 2024-04-16T11:37:54.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.990+0000 7fec56a00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec440692a0 0x7fec4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:54.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.990+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fec4c088190 con 0x7fec60102e60 2024-04-16T11:37:54.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.991+0000 7fec5ea00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec440692a0 0x7fec4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:54.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.991+0000 7fec5ea00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec440692a0 0x7fec4406b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec50004d60 tx=0x7fec5000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:54.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:54.995+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fec4c0547e0 con 0x7fec60102e60 2024-04-16T11:37:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: Detected new or changed devices on smithi045 2024-04-16T11:37:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi045 to 5314M 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:37:55.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:54 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:37:55.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.213+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec600fe570 con 0x7fec60102e60 2024-04-16T11:37:55.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.215+0000 7fec56a00700 1 -- 172.21.15.44:0/3878198779 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11031 (secure 0 0 0) 0x7fec4c057e00 con 0x7fec60102e60 2024-04-16T11:37:55.214 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:37:55.214 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":40,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:37:51.009429+0000","last_up_change":"2024-04-16T11:37:49.996185+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6803","nonce":1592689770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6805","nonce":1592689770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6809","nonce":1592689770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6807","nonce":1592689770}]},"public_addr":"172.21.15.44:6803/1592689770","cluster_addr":"172.21.15.44:6805/1592689770","heartbeat_back_addr":"172.21.15.44:6809/1592689770","heartbeat_front_addr":"172.21.15.44:6807/1592689770","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6811","nonce":412128623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6813","nonce":412128623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6817","nonce":412128623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6815","nonce":412128623}]},"public_addr":"172.21.15.44:6811/412128623","cluster_addr":"172.21.15.44:6813/412128623","heartbeat_back_addr":"172.21.15.44:6817/412128623","heartbeat_front_addr":"172.21.15.44:6815/412128623","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6819","nonce":2239266596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6821","nonce":2239266596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6825","nonce":2239266596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6823","nonce":2239266596}]},"public_addr":"172.21.15.44:6819/2239266596","cluster_addr":"172.21.15.44:6821/2239266596","heartbeat_back_addr":"172.21.15.44:6825/2239266596","heartbeat_front_addr":"172.21.15.44:6823/2239266596","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6801","nonce":3183628521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6803","nonce":3183628521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6807","nonce":3183628521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6805","nonce":3183628521}]},"public_addr":"172.21.15.45:6801/3183628521","cluster_addr":"172.21.15.45:6803/3183628521","heartbeat_back_addr":"172.21.15.45:6807/3183628521","heartbeat_front_addr":"172.21.15.45:6805/3183628521","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6809","nonce":884231764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6811","nonce":884231764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6815","nonce":884231764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6813","nonce":884231764}]},"public_addr":"172.21.15.45:6809/884231764","cluster_addr":"172.21.15.45:6811/884231764","heartbeat_back_addr":"172.21.15.45:6815/884231764","heartbeat_front_addr":"172.21.15.45:6813/884231764","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6817","nonce":1526857616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6819","nonce":1526857616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6823","nonce":1526857616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6821","nonce":1526857616}]},"public_addr":"172.21.15.45:6817/1526857616","cluster_addr":"172.21.15.45:6819/1526857616","heartbeat_back_addr":"172.21.15.45:6823/1526857616","heartbeat_front_addr":"172.21.15.45:6821/1526857616","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:37:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.217+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec440692a0 msgr2=0x7fec4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.217+0000 7fec65a00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec440692a0 0x7fec4406b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec50004d60 tx=0x7fec5000b040 comp rx=0 tx=0).stop 2024-04-16T11:37:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 msgr2=0x7fec6010bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:55.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 0x7fec6010bdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00b8b0 tx=0x7fec4c009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 shutdown_connections 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec440692a0 0x7fec4406b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec600656b0 0x7fec6010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 --2- 172.21.15.44:0/3878198779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec60102e60 0x7fec6010bdf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 >> 172.21.15.44:0/3878198779 conn(0x7fec60072bc0 msgr2=0x7fec600fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 shutdown_connections 2024-04-16T11:37:55.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:55.218+0000 7fec65a00700 1 -- 172.21.15.44:0/3878198779 wait complete. 2024-04-16T11:37:55.962 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:55 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3878198779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:37:56.039 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T11:36:12.538436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-16T11:37:56.039 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-16T11:37:56.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:55 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3878198779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:37:56.548 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:37:56.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:56 smithi044 ceph-mon[28731]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:57.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:56 smithi045 ceph-mon[36308]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:58.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 -- 172.21.15.44:0/2156811433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c105030 msgr2=0x7f129c105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:58.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 --2- 172.21.15.44:0/2156811433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c105030 0x7f129c105400 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f128c004940 tx=0x7f128c009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:58.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 -- 172.21.15.44:0/2156811433 shutdown_connections 2024-04-16T11:37:58.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 --2- 172.21.15.44:0/2156811433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c1059d0 0x7f129c0ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:58.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 --2- 172.21.15.44:0/2156811433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c105030 0x7f129c105400 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 -- 172.21.15.44:0/2156811433 >> 172.21.15.44:0/2156811433 conn(0x7f129c0fabd0 msgr2=0x7f129c0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 -- 172.21.15.44:0/2156811433 shutdown_connections 2024-04-16T11:37:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.697+0000 7f12a2000700 1 -- 172.21.15.44:0/2156811433 wait complete. 2024-04-16T11:37:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.699+0000 7f12a2000700 1 Processor -- start 2024-04-16T11:37:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.699+0000 7f12a2000700 1 -- start start 2024-04-16T11:37:58.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.699+0000 7f12a2000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c105030 0x7f129c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.699+0000 7f12a2000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 0x7f129c116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.699+0000 7f12a2000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c116780 con 0x7f129c1059d0 2024-04-16T11:37:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.699+0000 7f12a2000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c10fd60 con 0x7f129c105030 2024-04-16T11:37:58.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f1293e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 0x7f129c116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f1293e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 0x7f129c116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50986/0 (socket says 172.21.15.44:50986) 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f1293e00700 1 -- 172.21.15.44:0/4118771209 learned_addr learned my addr 172.21.15.44:0/4118771209 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f129b400700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c105030 0x7f129c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f1293e00700 1 -- 172.21.15.44:0/4118771209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c105030 msgr2=0x7f129c115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f1293e00700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c105030 0x7f129c115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.700+0000 7f1293e00700 1 -- 172.21.15.44:0/4118771209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c005040 con 0x7f129c1059d0 2024-04-16T11:37:58.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.701+0000 7f1293e00700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 0x7f129c116130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1284002a80 tx=0x7f1284004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.701+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1284004e80 con 0x7f129c1059d0 2024-04-16T11:37:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.701+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f128400f040 con 0x7f129c1059d0 2024-04-16T11:37:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.701+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f128400daa0 con 0x7f129c1059d0 2024-04-16T11:37:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.701+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c110040 con 0x7f129c1059d0 2024-04-16T11:37:58.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.701+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c110600 con 0x7f129c1059d0 2024-04-16T11:37:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.703+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c100780 con 0x7f129c1059d0 2024-04-16T11:37:58.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.704+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1284007cc0 con 0x7f129c1059d0 2024-04-16T11:37:58.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.704+0000 7f1293400700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f12800692a0 0x7f128006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:37:58.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.705+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f12840863d0 con 0x7f129c1059d0 2024-04-16T11:37:58.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.705+0000 7f129b400700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f12800692a0 0x7f128006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:37:58.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.705+0000 7f129b400700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f12800692a0 0x7f128006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f128c00b3b0 tx=0x7f128c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:37:58.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.709+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f128404d250 con 0x7f129c1059d0 2024-04-16T11:37:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.923+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f129c04c710 con 0x7f129c1059d0 2024-04-16T11:37:58.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.924+0000 7f1293400700 1 -- 172.21.15.44:0/4118771209 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12840564d0 con 0x7f129c1059d0 2024-04-16T11:37:58.923 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.928+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f12800692a0 msgr2=0x7f128006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.928+0000 7f12a2000700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f12800692a0 0x7f128006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f128c00b3b0 tx=0x7f128c009e30 comp rx=0 tx=0).stop 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.928+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 msgr2=0x7f129c116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.928+0000 7f12a2000700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 0x7f129c116130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1284002a80 tx=0x7f1284004630 comp rx=0 tx=0).stop 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 shutdown_connections 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f12800692a0 0x7f128006b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:58.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c105030 0x7f129c115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:58.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 --2- 172.21.15.44:0/4118771209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c1059d0 0x7f129c116130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:37:58.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 >> 172.21.15.44:0/4118771209 conn(0x7f129c0fabd0 msgr2=0x7f129c0fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:37:58.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 shutdown_connections 2024-04-16T11:37:58.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:37:58.929+0000 7f12a2000700 1 -- 172.21.15.44:0/4118771209 wait complete. 2024-04-16T11:37:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:58 smithi044 ceph-mon[28731]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:37:58 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/4118771209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:37:59.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:58 smithi045 ceph-mon[36308]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:37:59.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:37:58 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/4118771209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:37:59.729 INFO:tasks.cephadm:Setting up client nodes... 2024-04-16T11:37:59.729 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T11:38:00.252 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:01.224 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:00 smithi044 ceph-mon[28731]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:01.224 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:00 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:00 smithi045 ceph-mon[36308]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:00 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 -- 172.21.15.44:0/1218227544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 msgr2=0x7fd38c0fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 --2- 172.21.15.44:0/1218227544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c0fe350 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd378004940 tx=0x7fd378009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 -- 172.21.15.44:0/1218227544 shutdown_connections 2024-04-16T11:38:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 --2- 172.21.15.44:0/1218227544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c0fe350 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:02.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 --2- 172.21.15.44:0/1218227544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 0x7fd38c0fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 -- 172.21.15.44:0/1218227544 >> 172.21.15.44:0/1218227544 conn(0x7fd38c0f8370 msgr2=0x7fd38c0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 -- 172.21.15.44:0/1218227544 shutdown_connections 2024-04-16T11:38:02.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.524+0000 7fd392400700 1 -- 172.21.15.44:0/1218227544 wait complete. 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.525+0000 7fd392400700 1 Processor -- start 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.525+0000 7fd392400700 1 -- start start 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd392400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 0x7fd38c10fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd392400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c10ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd392400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c110570 con 0x7fd38c0fdee0 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd392400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c110dc0 con 0x7fd38c0fcc30 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd38be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c10ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd38be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c10ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41254/0 (socket says 172.21.15.44:41254) 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd38be00700 1 -- 172.21.15.44:0/671109056 learned_addr learned my addr 172.21.15.44:0/671109056 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.526+0000 7fd391000700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 0x7fd38c10fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.527+0000 7fd38be00700 1 -- 172.21.15.44:0/671109056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 msgr2=0x7fd38c10fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:02.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.527+0000 7fd38be00700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 0x7fd38c10fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.527+0000 7fd38be00700 1 -- 172.21.15.44:0/671109056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378005040 con 0x7fd38c0fdee0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.527+0000 7fd391000700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 0x7fd38c10fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.527+0000 7fd38be00700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c10ff70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd378013870 tx=0x7fd378009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.527+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd37800e070 con 0x7fd38c0fdee0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.528+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd37801c400 con 0x7fd38c0fdee0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.528+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c111040 con 0x7fd38c0fdee0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.528+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd37800b630 con 0x7fd38c0fdee0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.528+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c111520 con 0x7fd38c0fdee0 2024-04-16T11:38:02.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.529+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c101870 con 0x7fd38c0fdee0 2024-04-16T11:38:02.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.530+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd37801c570 con 0x7fd38c0fdee0 2024-04-16T11:38:02.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.531+0000 7fd388c00700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd3740692a0 0x7fd37406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:02.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.531+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd3780886c0 con 0x7fd38c0fdee0 2024-04-16T11:38:02.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.531+0000 7fd391000700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd3740692a0 0x7fd37406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:02.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.532+0000 7fd391000700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd3740692a0 0x7fd37406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd380004600 tx=0x7fd38000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:02.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.536+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd378058720 con 0x7fd38c0fdee0 2024-04-16T11:38:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.835+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd38c0fc030 con 0x7fd38c0fdee0 2024-04-16T11:38:02.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.842+0000 7fd388c00700 1 -- 172.21.15.44:0/671109056 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd3780582b0 con 0x7fd38c0fdee0 2024-04-16T11:38:02.841 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-16T11:38:02.841 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQAaYx5mAQT1MRAAbOgWAc5EPtKk6ViYPGDwWQ== 2024-04-16T11:38:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd3740692a0 msgr2=0x7fd37406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd3740692a0 0x7fd37406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd380004600 tx=0x7fd38000b040 comp rx=0 tx=0).stop 2024-04-16T11:38:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 msgr2=0x7fd38c10ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c10ff70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd378013870 tx=0x7fd378009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 shutdown_connections 2024-04-16T11:38:02.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd3740692a0 0x7fd37406b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0fcc30 0x7fd38c10fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 --2- 172.21.15.44:0/671109056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd38c0fdee0 0x7fd38c10ff70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 >> 172.21.15.44:0/671109056 conn(0x7fd38c0f8370 msgr2=0x7fd38c0f9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 shutdown_connections 2024-04-16T11:38:02.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:02.845+0000 7fd392400700 1 -- 172.21.15.44:0/671109056 wait complete. 2024-04-16T11:38:03.137 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:02 smithi044 ceph-mon[28731]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:03.137 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:02 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/671109056' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T11:38:03.137 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:02 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/671109056' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T11:38:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:02 smithi045 ceph-mon[36308]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:03.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:02 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/671109056' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T11:38:03.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:02 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/671109056' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T11:38:03.606 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:38:03.607 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-16T11:38:03.607 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T11:38:03.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T11:38:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi045/config 2024-04-16T11:38:05.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:04 smithi045 ceph-mon[36308]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:04 smithi044 ceph-mon[28731]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 -- 172.21.15.45:0/458978624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fce70 msgr2=0x7fe1600fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 --2- 172.21.15.45:0/458978624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fce70 0x7fe1600fd280 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe150004940 tx=0x7fe150009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 -- 172.21.15.45:0/458978624 shutdown_connections 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 --2- 172.21.15.45:0/458978624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1600fe070 0x7fe1600fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 --2- 172.21.15.45:0/458978624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fce70 0x7fe1600fd280 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 -- 172.21.15.45:0/458978624 >> 172.21.15.45:0/458978624 conn(0x7fe1600f8540 msgr2=0x7fe1600fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 -- 172.21.15.45:0/458978624 shutdown_connections 2024-04-16T11:38:06.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.418+0000 7fe165800700 1 -- 172.21.15.45:0/458978624 wait complete. 2024-04-16T11:38:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.419+0000 7fe165800700 1 Processor -- start 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe165800700 1 -- start start 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe165800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1600fce70 0x7fe16010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe165800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 0x7fe1601100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe165800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1601106b0 con 0x7fe1600fce70 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe165800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe160110820 con 0x7fe1600fe070 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe156000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 0x7fe1601100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe156000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 0x7fe1601100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57104/0 (socket says 172.21.15.45:57104) 2024-04-16T11:38:06.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.420+0000 7fe156000700 1 -- 172.21.15.45:0/1051576577 learned_addr learned my addr 172.21.15.45:0/1051576577 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-16T11:38:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe15ea00700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1600fce70 0x7fe16010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe156000700 1 -- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1600fce70 msgr2=0x7fe16010fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe156000700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1600fce70 0x7fe16010fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe156000700 1 -- 172.21.15.45:0/1051576577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe150005040 con 0x7fe1600fe070 2024-04-16T11:38:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe156000700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 0x7fe1601100b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe148002a50 tx=0x7fe1480046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:06.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe148007cc0 con 0x7fe1600fe070 2024-04-16T11:38:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe14800f040 con 0x7fe1600fe070 2024-04-16T11:38:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe160115280 con 0x7fe1600fe070 2024-04-16T11:38:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.421+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe148009640 con 0x7fe1600fe070 2024-04-16T11:38:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.422+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe160115840 con 0x7fe1600fe070 2024-04-16T11:38:06.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.423+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe160101a00 con 0x7fe1600fe070 2024-04-16T11:38:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.424+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe1480097a0 con 0x7fe1600fe070 2024-04-16T11:38:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.425+0000 7fe157400700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe1440692a0 0x7fe14406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.425+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe148086c20 con 0x7fe1600fe070 2024-04-16T11:38:06.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.425+0000 7fe15ea00700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe1440692a0 0x7fe14406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.427+0000 7fe15ea00700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe1440692a0 0x7fe14406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe15000b970 tx=0x7fe150009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:06.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.430+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe148056c80 con 0x7fe1600fe070 2024-04-16T11:38:06.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.730+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe16004c710 con 0x7fe1600fe070 2024-04-16T11:38:06.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.741+0000 7fe157400700 1 -- 172.21.15.45:0/1051576577 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe148056810 con 0x7fe1600fe070 2024-04-16T11:38:06.739 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-04-16T11:38:06.739 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAeYx5m7462KxAAE+V15683MTw/rsPQ0jIH/Q== 2024-04-16T11:38:06.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.744+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe1440692a0 msgr2=0x7fe14406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:06.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.744+0000 7fe165800700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe1440692a0 0x7fe14406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe15000b970 tx=0x7fe150009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.744+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 msgr2=0x7fe1601100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.744+0000 7fe165800700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 0x7fe1601100b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe148002a50 tx=0x7fe1480046b0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 shutdown_connections 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe1440692a0 0x7fe14406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe1600fce70 0x7fe16010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 --2- 172.21.15.45:0/1051576577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1600fe070 0x7fe1601100b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 >> 172.21.15.45:0/1051576577 conn(0x7fe1600f8540 msgr2=0x7fe1600f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 shutdown_connections 2024-04-16T11:38:06.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:38:06.745+0000 7fe165800700 1 -- 172.21.15.45:0/1051576577 wait complete. 2024-04-16T11:38:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:06 smithi045 ceph-mon[36308]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:06 smithi045 ceph-mon[36308]: from='client.? 172.21.15.45:0/1051576577' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T11:38:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:06 smithi045 ceph-mon[36308]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T11:38:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:06 smithi045 ceph-mon[36308]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T11:38:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:06 smithi044 ceph-mon[28731]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:06 smithi044 ceph-mon[28731]: from='client.? 172.21.15.45:0/1051576577' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T11:38:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:06 smithi044 ceph-mon[28731]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T11:38:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:06 smithi044 ceph-mon[28731]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T11:38:07.572 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-16T11:38:07.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-16T11:38:07.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T11:38:07.742 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T11:38:07.742 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-16T11:38:07.742 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-16T11:38:08.256 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:08 smithi044 ceph-mon[28731]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:08 smithi045 ceph-mon[36308]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:10.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.638+0000 7f58ef200700 1 -- 172.21.15.44:0/2855775923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 msgr2=0x7f58f0103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:10.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.638+0000 7f58ef200700 1 --2- 172.21.15.44:0/2855775923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f0103330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58dc004940 tx=0x7f58dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:10.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.638+0000 7f58ef200700 1 -- 172.21.15.44:0/2855775923 shutdown_connections 2024-04-16T11:38:10.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.638+0000 7f58ef200700 1 --2- 172.21.15.44:0/2855775923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f0103330 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:10.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.638+0000 7f58ef200700 1 --2- 172.21.15.44:0/2855775923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 0x7f58f0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:10.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.638+0000 7f58ef200700 1 -- 172.21.15.44:0/2855775923 >> 172.21.15.44:0/2855775923 conn(0x7f58f0072a60 msgr2=0x7f58f0074e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:10.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.639+0000 7f58ef200700 1 -- 172.21.15.44:0/2855775923 shutdown_connections 2024-04-16T11:38:10.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.639+0000 7f58ef200700 1 -- 172.21.15.44:0/2855775923 wait complete. 2024-04-16T11:38:10.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.639+0000 7f58ef200700 1 Processor -- start 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ef200700 1 -- start start 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ef200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 0x7f58f011a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ef200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f011a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ef200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f011aba0 con 0x7f58f0109c20 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ef200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f0113f70 con 0x7f58f0109280 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ed400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f011a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ed400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f011a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41280/0 (socket says 172.21.15.44:41280) 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.640+0000 7f58ed400700 1 -- 172.21.15.44:0/2466207661 learned_addr learned my addr 172.21.15.44:0/2466207661 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58ede00700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 0x7f58f011a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58ed400700 1 -- 172.21.15.44:0/2466207661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 msgr2=0x7f58f011a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58ed400700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 0x7f58f011a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58ed400700 1 -- 172.21.15.44:0/2466207661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc005040 con 0x7f58f0109c20 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58ede00700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 0x7f58f011a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:38:10.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58ed400700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f011a550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009100 tx=0x7f58dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.641+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58dc00e040 con 0x7f58f0109c20 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.642+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58dc013420 con 0x7f58f0109c20 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.642+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f01141f0 con 0x7f58f0109c20 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.642+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58dc01c630 con 0x7f58f0109c20 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.642+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f01147b0 con 0x7f58f0109c20 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.643+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f01049d0 con 0x7f58f0109c20 2024-04-16T11:38:10.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.644+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f58dc013590 con 0x7f58f0109c20 2024-04-16T11:38:10.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.645+0000 7f58e6000700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f58d40692a0 0x7f58d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:10.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.645+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58dc088170 con 0x7f58f0109c20 2024-04-16T11:38:10.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.645+0000 7f58ede00700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f58d40692a0 0x7f58d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:10.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.646+0000 7f58ede00700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f58d40692a0 0x7f58d406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009fd0 tx=0x7f58e000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:10.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.649+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f58dc0581d0 con 0x7f58f0109c20 2024-04-16T11:38:10.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.925+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f58f0076590 con 0x7f58f0109c20 2024-04-16T11:38:10.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.935+0000 7f58e6000700 1 -- 172.21.15.44:0/2466207661 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167178 (secure 0 0 0) 0x7f58dc057d60 con 0x7f58f0109c20 2024-04-16T11:38:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f58d40692a0 msgr2=0x7f58d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f58d40692a0 0x7f58d406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009fd0 tx=0x7f58e000b040 comp rx=0 tx=0).stop 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 msgr2=0x7f58f011a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f011a550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009100 tx=0x7f58dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 shutdown_connections 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f58d40692a0 0x7f58d406b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58f0109280 0x7f58f011a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 --2- 172.21.15.44:0/2466207661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58f0109c20 0x7f58f011a550 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 >> 172.21.15.44:0/2466207661 conn(0x7f58f0072a60 msgr2=0x7f58f00749b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 shutdown_connections 2024-04-16T11:38:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:10.939+0000 7f58ef200700 1 -- 172.21.15.44:0/2466207661 wait complete. 2024-04-16T11:38:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:10 smithi044 ceph-mon[28731]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:11 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2466207661' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T11:38:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:11 smithi045 ceph-mon[36308]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:11 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2466207661' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T11:38:11.782 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":19,"flags":0,"active_gid":14212,"active_name":"smithi044.sczobv","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2086828317},{"type":"v1","addr":"172.21.15.44:6801","nonce":2086828317}]},"active_addr":"172.21.15.44:6801/2086828317","active_change":"2024-04-16T11:33:15.828543+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi045.sdlwif","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.44:8443/","prometheus":"http://172.21.15.44:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1916760620}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":3788212463}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":567559671}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1480410906}]}]} 2024-04-16T11:38:11.786 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-16T11:38:11.787 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-16T11:38:11.787 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T11:38:12.303 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:13.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:13 smithi044 ceph-mon[28731]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:13.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:13 smithi045 ceph-mon[36308]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 -- 172.21.15.44:0/1815795224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 msgr2=0x7f4ea41020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 --2- 172.21.15.44:0/1815795224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea41020e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e98004940 tx=0x7f4e98009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 -- 172.21.15.44:0/1815795224 shutdown_connections 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 --2- 172.21.15.44:0/1815795224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 0x7f4ea40fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 --2- 172.21.15.44:0/1815795224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea41020e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 -- 172.21.15.44:0/1815795224 >> 172.21.15.44:0/1815795224 conn(0x7f4ea4072bc0 msgr2=0x7f4ea4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:14.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 -- 172.21.15.44:0/1815795224 shutdown_connections 2024-04-16T11:38:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.482+0000 7f4eab600700 1 -- 172.21.15.44:0/1815795224 wait complete. 2024-04-16T11:38:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.483+0000 7f4eab600700 1 Processor -- start 2024-04-16T11:38:14.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.483+0000 7f4eab600700 1 -- start start 2024-04-16T11:38:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.483+0000 7f4eab600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea4113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:14.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.483+0000 7f4eab600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 0x7f4ea41140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4eab600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4114730 con 0x7f4ea4101d10 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4eab600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea410db20 con 0x7f4ea41026b0 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea4113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea4113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46976/0 (socket says 172.21.15.44:46976) 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea8c00700 1 -- 172.21.15.44:0/4013686925 learned_addr learned my addr 172.21.15.44:0/4013686925 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea8c00700 1 -- 172.21.15.44:0/4013686925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 msgr2=0x7f4ea41140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea3e00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 0x7f4ea41140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea8c00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 0x7f4ea41140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea8c00700 1 -- 172.21.15.44:0/4013686925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98005040 con 0x7f4ea4101d10 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.484+0000 7f4ea3e00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 0x7f4ea41140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.485+0000 7f4ea8c00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea4113ba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e9800b8b0 tx=0x7f4e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.485+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9800e040 con 0x7f4ea4101d10 2024-04-16T11:38:14.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.485+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e98004290 con 0x7f4ea4101d10 2024-04-16T11:38:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.485+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9801c3f0 con 0x7f4ea4101d10 2024-04-16T11:38:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.485+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea410dda0 con 0x7f4ea4101d10 2024-04-16T11:38:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.485+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea410e360 con 0x7f4ea4101d10 2024-04-16T11:38:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.486+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea40fd460 con 0x7f4ea4101d10 2024-04-16T11:38:14.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.487+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f4e980137b0 con 0x7f4ea4101d10 2024-04-16T11:38:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.488+0000 7f4ea0c00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4e8c0692a0 0x7f4e8c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.488+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4e980880a0 con 0x7f4ea4101d10 2024-04-16T11:38:14.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.488+0000 7f4ea3e00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4e8c0692a0 0x7f4e8c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:14.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.489+0000 7f4ea3e00700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4e8c0692a0 0x7f4e8c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e90007fd0 tx=0x7f4e90009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:14.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.493+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4e98058240 con 0x7f4ea4101d10 2024-04-16T11:38:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.703+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ea4108e10 con 0x7f4ea4101d10 2024-04-16T11:38:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.705+0000 7f4ea0c00700 1 -- 172.21.15.44:0/4013686925 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11031 (secure 0 0 0) 0x7f4e9806ae60 con 0x7f4ea4101d10 2024-04-16T11:38:14.703 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:38:14.703 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":40,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:37:51.009429+0000","last_up_change":"2024-04-16T11:37:49.996185+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6803","nonce":1592689770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6805","nonce":1592689770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6809","nonce":1592689770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6807","nonce":1592689770}]},"public_addr":"172.21.15.44:6803/1592689770","cluster_addr":"172.21.15.44:6805/1592689770","heartbeat_back_addr":"172.21.15.44:6809/1592689770","heartbeat_front_addr":"172.21.15.44:6807/1592689770","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6811","nonce":412128623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6813","nonce":412128623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6817","nonce":412128623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6815","nonce":412128623}]},"public_addr":"172.21.15.44:6811/412128623","cluster_addr":"172.21.15.44:6813/412128623","heartbeat_back_addr":"172.21.15.44:6817/412128623","heartbeat_front_addr":"172.21.15.44:6815/412128623","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6819","nonce":2239266596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6821","nonce":2239266596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6825","nonce":2239266596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6823","nonce":2239266596}]},"public_addr":"172.21.15.44:6819/2239266596","cluster_addr":"172.21.15.44:6821/2239266596","heartbeat_back_addr":"172.21.15.44:6825/2239266596","heartbeat_front_addr":"172.21.15.44:6823/2239266596","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6801","nonce":3183628521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6803","nonce":3183628521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6807","nonce":3183628521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6805","nonce":3183628521}]},"public_addr":"172.21.15.45:6801/3183628521","cluster_addr":"172.21.15.45:6803/3183628521","heartbeat_back_addr":"172.21.15.45:6807/3183628521","heartbeat_front_addr":"172.21.15.45:6805/3183628521","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6809","nonce":884231764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6811","nonce":884231764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6815","nonce":884231764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6813","nonce":884231764}]},"public_addr":"172.21.15.45:6809/884231764","cluster_addr":"172.21.15.45:6811/884231764","heartbeat_back_addr":"172.21.15.45:6815/884231764","heartbeat_front_addr":"172.21.15.45:6813/884231764","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6817","nonce":1526857616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6819","nonce":1526857616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6823","nonce":1526857616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6821","nonce":1526857616}]},"public_addr":"172.21.15.45:6817/1526857616","cluster_addr":"172.21.15.45:6819/1526857616","heartbeat_back_addr":"172.21.15.45:6823/1526857616","heartbeat_front_addr":"172.21.15.45:6821/1526857616","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:38:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.707+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4e8c0692a0 msgr2=0x7f4e8c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.707+0000 7f4eab600700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4e8c0692a0 0x7f4e8c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e90007fd0 tx=0x7f4e90009040 comp rx=0 tx=0).stop 2024-04-16T11:38:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.707+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 msgr2=0x7f4ea4113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea4113ba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e9800b8b0 tx=0x7f4e98009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 shutdown_connections 2024-04-16T11:38:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4e8c0692a0 0x7f4e8c06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ea4101d10 0x7f4ea4113ba0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 --2- 172.21.15.44:0/4013686925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ea41026b0 0x7f4ea41140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 >> 172.21.15.44:0/4013686925 conn(0x7f4ea4072bc0 msgr2=0x7f4ea40fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 shutdown_connections 2024-04-16T11:38:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:14.708+0000 7f4eab600700 1 -- 172.21.15.44:0/4013686925 wait complete. 2024-04-16T11:38:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:15 smithi045 ceph-mon[36308]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:15.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:15 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/4013686925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:38:15.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:15 smithi044 ceph-mon[28731]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:15.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:15 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/4013686925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:38:15.546 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-16T11:38:15.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T11:38:16.052 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:16.379 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:17 smithi045 ceph-mon[36308]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:17.406 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:17 smithi044 ceph-mon[28731]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 -- 172.21.15.44:0/1709905559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 msgr2=0x7f61c40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 --2- 172.21.15.44:0/1709905559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c40fce30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f61b0004940 tx=0x7f61b0009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 -- 172.21.15.44:0/1709905559 shutdown_connections 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 --2- 172.21.15.44:0/1709905559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c40fce30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 --2- 172.21.15.44:0/1709905559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c4102e20 0x7f61c41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 -- 172.21.15.44:0/1709905559 >> 172.21.15.44:0/1709905559 conn(0x7f61c40f89b0 msgr2=0x7f61c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 -- 172.21.15.44:0/1709905559 shutdown_connections 2024-04-16T11:38:18.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.260+0000 7f61c9000700 1 -- 172.21.15.44:0/1709905559 wait complete. 2024-04-16T11:38:18.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.261+0000 7f61c9000700 1 Processor -- start 2024-04-16T11:38:18.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c9000700 1 -- start start 2024-04-16T11:38:18.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c9000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c4102e20 0x7f61c4115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:18.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c4116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:18.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4116760 con 0x7f61c41037c0 2024-04-16T11:38:18.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c9000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c410fd80 con 0x7f61c4102e20 2024-04-16T11:38:18.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c4116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:18.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c4116110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46982/0 (socket says 172.21.15.44:46982) 2024-04-16T11:38:18.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.262+0000 7f61c1400700 1 -- 172.21.15.44:0/2786663830 learned_addr learned my addr 172.21.15.44:0/2786663830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:18.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.263+0000 7f61c1e00700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c4102e20 0x7f61c4115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.263+0000 7f61c1400700 1 -- 172.21.15.44:0/2786663830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c4102e20 msgr2=0x7f61c4115bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.263+0000 7f61c1400700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c4102e20 0x7f61c4115bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.263+0000 7f61c1400700 1 -- 172.21.15.44:0/2786663830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0005040 con 0x7f61c41037c0 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.263+0000 7f61c1400700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c4116110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009100 tx=0x7f61b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.263+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61b000e040 con 0x7f61c41037c0 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.264+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61b0004290 con 0x7f61c41037c0 2024-04-16T11:38:18.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.264+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61b001c3f0 con 0x7f61c41037c0 2024-04-16T11:38:18.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.264+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c4110000 con 0x7f61c41037c0 2024-04-16T11:38:18.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.264+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c4110540 con 0x7f61c41037c0 2024-04-16T11:38:18.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.265+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c4102140 con 0x7f61c41037c0 2024-04-16T11:38:18.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.266+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f61b0013790 con 0x7f61c41037c0 2024-04-16T11:38:18.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.267+0000 7f61ba000700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f61a80692a0 0x7f61a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:18.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.267+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f61b00880f0 con 0x7f61c41037c0 2024-04-16T11:38:18.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.267+0000 7f61c1e00700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f61a80692a0 0x7f61a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:18.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.268+0000 7f61c1e00700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f61a80692a0 0x7f61a806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f61b4004990 tx=0x7f61b4009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.272+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f61b0054740 con 0x7f61c41037c0 2024-04-16T11:38:18.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.488+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61c4111260 con 0x7f61c41037c0 2024-04-16T11:38:18.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.490+0000 7f61ba000700 1 -- 172.21.15.44:0/2786663830 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11031 (secure 0 0 0) 0x7f61b0013420 con 0x7f61c41037c0 2024-04-16T11:38:18.489 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:38:18.489 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":40,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:37:51.009429+0000","last_up_change":"2024-04-16T11:37:49.996185+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6803","nonce":1592689770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6805","nonce":1592689770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6809","nonce":1592689770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6807","nonce":1592689770}]},"public_addr":"172.21.15.44:6803/1592689770","cluster_addr":"172.21.15.44:6805/1592689770","heartbeat_back_addr":"172.21.15.44:6809/1592689770","heartbeat_front_addr":"172.21.15.44:6807/1592689770","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6811","nonce":412128623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6813","nonce":412128623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6817","nonce":412128623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6815","nonce":412128623}]},"public_addr":"172.21.15.44:6811/412128623","cluster_addr":"172.21.15.44:6813/412128623","heartbeat_back_addr":"172.21.15.44:6817/412128623","heartbeat_front_addr":"172.21.15.44:6815/412128623","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6819","nonce":2239266596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6821","nonce":2239266596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6825","nonce":2239266596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6823","nonce":2239266596}]},"public_addr":"172.21.15.44:6819/2239266596","cluster_addr":"172.21.15.44:6821/2239266596","heartbeat_back_addr":"172.21.15.44:6825/2239266596","heartbeat_front_addr":"172.21.15.44:6823/2239266596","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6801","nonce":3183628521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6803","nonce":3183628521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6807","nonce":3183628521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6805","nonce":3183628521}]},"public_addr":"172.21.15.45:6801/3183628521","cluster_addr":"172.21.15.45:6803/3183628521","heartbeat_back_addr":"172.21.15.45:6807/3183628521","heartbeat_front_addr":"172.21.15.45:6805/3183628521","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6809","nonce":884231764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6811","nonce":884231764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6815","nonce":884231764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6813","nonce":884231764}]},"public_addr":"172.21.15.45:6809/884231764","cluster_addr":"172.21.15.45:6811/884231764","heartbeat_back_addr":"172.21.15.45:6815/884231764","heartbeat_front_addr":"172.21.15.45:6813/884231764","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6817","nonce":1526857616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6819","nonce":1526857616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6823","nonce":1526857616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6821","nonce":1526857616}]},"public_addr":"172.21.15.45:6817/1526857616","cluster_addr":"172.21.15.45:6819/1526857616","heartbeat_back_addr":"172.21.15.45:6823/1526857616","heartbeat_front_addr":"172.21.15.45:6821/1526857616","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:38:18.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f61a80692a0 msgr2=0x7f61a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:18.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f61a80692a0 0x7f61a806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f61b4004990 tx=0x7f61b4009ee0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 msgr2=0x7f61c4116110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:18.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c4116110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009100 tx=0x7f61b0009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:18.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 shutdown_connections 2024-04-16T11:38:18.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f61a80692a0 0x7f61a806b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c4102e20 0x7f61c4115bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 --2- 172.21.15.44:0/2786663830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61c41037c0 0x7f61c4116110 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 >> 172.21.15.44:0/2786663830 conn(0x7f61c40f89b0 msgr2=0x7f61c4070b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 shutdown_connections 2024-04-16T11:38:18.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:18.494+0000 7f61c9000700 1 -- 172.21.15.44:0/2786663830 wait complete. 2024-04-16T11:38:19.195 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:19 smithi044 ceph-mon[28731]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:19.195 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:19 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2786663830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:38:19.250 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-16T11:38:19.251 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-16T11:38:19.251 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-16T11:38:19.252 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-16T11:38:19.252 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-16T11:38:19.253 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-16T11:38:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:19 smithi045 ceph-mon[36308]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:19 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2786663830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:38:19.807 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:19.815 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:21.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:21 smithi045 ceph-mon[36308]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:21.400 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:21 smithi044 ceph-mon[28731]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:21.523 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:21.525 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:22.397 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:23 smithi044 ceph-mon[28731]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:23.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:23 smithi045 ceph-mon[36308]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:24.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.451+0000 7f0d14c00700 1 -- 172.21.15.44:0/1439319282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fbb80 msgr2=0x7f0d100fbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:24.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.451+0000 7f0d14c00700 1 --2- 172.21.15.44:0/1439319282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fbb80 0x7f0d100fbf50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d00004940 tx=0x7f0d00009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.452+0000 7f0d14c00700 1 -- 172.21.15.44:0/1439319282 shutdown_connections 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.452+0000 7f0d14c00700 1 --2- 172.21.15.44:0/1439319282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d100fc490 0x7f0d101091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.452+0000 7f0d14c00700 1 --2- 172.21.15.44:0/1439319282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fbb80 0x7f0d100fbf50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.452+0000 7f0d14c00700 1 -- 172.21.15.44:0/1439319282 >> 172.21.15.44:0/1439319282 conn(0x7f0d100f7870 msgr2=0x7f0d100f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.452+0000 7f0d14c00700 1 -- 172.21.15.44:0/1439319282 shutdown_connections 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.452+0000 7f0d14c00700 1 -- 172.21.15.44:0/1439319282 wait complete. 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.453+0000 7f0d14c00700 1 Processor -- start 2024-04-16T11:38:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.453+0000 7f0d14c00700 1 -- start start 2024-04-16T11:38:24.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d14c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d100fbb80 0x7f0d10115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:24.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d14c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 0x7f0d10116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:24.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d14c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d101166e0 con 0x7f0d100fc490 2024-04-16T11:38:24.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d14c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1010fcc0 con 0x7f0d100fbb80 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d0d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 0x7f0d10116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d0d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 0x7f0d10116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45492/0 (socket says 172.21.15.44:45492) 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.454+0000 7f0d0d400700 1 -- 172.21.15.44:0/3232983691 learned_addr learned my addr 172.21.15.44:0/3232983691 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d0de00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d100fbb80 0x7f0d10115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d0d400700 1 -- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d100fbb80 msgr2=0x7f0d10115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d0d400700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d100fbb80 0x7f0d10115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d0d400700 1 -- 172.21.15.44:0/3232983691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00005040 con 0x7f0d100fc490 2024-04-16T11:38:24.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d0d400700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 0x7f0d10116090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc002a50 tx=0x7f0cfc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:24.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cfc007cc0 con 0x7f0d100fc490 2024-04-16T11:38:24.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1010ffa0 con 0x7f0d100fc490 2024-04-16T11:38:24.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cfc00f040 con 0x7f0d100fc490 2024-04-16T11:38:24.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.455+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cfc009640 con 0x7f0d100fc490 2024-04-16T11:38:24.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.456+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d10110560 con 0x7f0d100fc490 2024-04-16T11:38:24.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.456+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d10110a40 con 0x7f0d100fc490 2024-04-16T11:38:24.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.458+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0cfc0097a0 con 0x7f0d100fc490 2024-04-16T11:38:24.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.459+0000 7f0d06000700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0cf40692a0 0x7f0cf406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:24.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.459+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0cfc086d30 con 0x7f0d100fc490 2024-04-16T11:38:24.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.459+0000 7f0d0de00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0cf40692a0 0x7f0cf406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:24.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.459+0000 7f0d06000700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] conn(0x7f0cf406ef70 0x7f0cf4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:24.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.459+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 --> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cf4071a30 con 0x7f0cf406ef70 2024-04-16T11:38:24.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.459+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cfc08c070 con 0x7f0d100fc490 2024-04-16T11:38:24.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.460+0000 7f0d0de00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0cf40692a0 0x7f0cf406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d00004f90 tx=0x7f0d00009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:24.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.460+0000 7f0d0e800700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] conn(0x7f0cf406ef70 0x7f0cf4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:24.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.461+0000 7f0d0e800700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] conn(0x7f0cf406ef70 0x7f0cf4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:24.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.463+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== osd.3 v2:172.21.15.45:6800/3183628521 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0cf4071a30 con 0x7f0cf406ef70 2024-04-16T11:38:24.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.480+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 --> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d100f8610 con 0x7f0cf406ef70 2024-04-16T11:38:24.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.481+0000 7f0d06000700 1 -- 172.21.15.44:0/3232983691 <== osd.3 v2:172.21.15.45:6800/3183628521 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0d100f8610 con 0x7f0cf406ef70 2024-04-16T11:38:24.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] conn(0x7f0cf406ef70 msgr2=0x7f0cf4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:24.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] conn(0x7f0cf406ef70 0x7f0cf4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0cf40692a0 msgr2=0x7f0cf406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:24.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0cf40692a0 0x7f0cf406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d00004f90 tx=0x7f0d00009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:24.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 msgr2=0x7f0d10116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:24.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 0x7f0d10116090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc002a50 tx=0x7f0cfc0046b0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 shutdown_connections 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:6800/3183628521,v1:172.21.15.45:6801/3183628521] conn(0x7f0cf406ef70 0x7f0cf4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0cf40692a0 0x7f0cf406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d100fbb80 0x7f0d10115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 --2- 172.21.15.44:0/3232983691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d100fc490 0x7f0d10116090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 >> 172.21.15.44:0/3232983691 conn(0x7f0d100f7870 msgr2=0x7f0d100f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 shutdown_connections 2024-04-16T11:38:24.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:24.482+0000 7f0d14c00700 1 -- 172.21.15.44:0/3232983691 wait complete. 2024-04-16T11:38:25.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:25 smithi044 ceph-mon[28731]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:25.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:25 smithi045 ceph-mon[36308]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:25.650 INFO:teuthology.orchestra.run.smithi044.stdout:111669149717 2024-04-16T11:38:25.651 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-16T11:38:26.685 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:26 smithi044 ceph-mon[28731]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:26 smithi045 ceph-mon[36308]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.364+0000 7fa7a8a00700 1 -- 172.21.15.44:0/921249711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4065bf0 msgr2=0x7fa7a4066060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.364+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/921249711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4065bf0 0x7fa7a4066060 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa78c004940 tx=0x7fa78c009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.364+0000 7fa7a8a00700 1 -- 172.21.15.44:0/921249711 shutdown_connections 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.364+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/921249711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4065bf0 0x7fa7a4066060 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.364+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/921249711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7a4101d40 0x7fa7a40656b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.364+0000 7fa7a8a00700 1 -- 172.21.15.44:0/921249711 >> 172.21.15.44:0/921249711 conn(0x7fa7a4072bc0 msgr2=0x7fa7a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.365+0000 7fa7a8a00700 1 -- 172.21.15.44:0/921249711 shutdown_connections 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.365+0000 7fa7a8a00700 1 -- 172.21.15.44:0/921249711 wait complete. 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.366+0000 7fa7a8a00700 1 Processor -- start 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.366+0000 7fa7a8a00700 1 -- start start 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa7a8a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7a4065bf0 0x7fa7a411a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa7a8a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 0x7fa7a411a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa7a8a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a411ad60 con 0x7fa7a4101d40 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa7a8a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4114130 con 0x7fa7a4065bf0 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa799400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 0x7fa7a411a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa799400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 0x7fa7a411a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45508/0 (socket says 172.21.15.44:45508) 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa799400700 1 -- 172.21.15.44:0/410224365 learned_addr learned my addr 172.21.15.44:0/410224365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.367+0000 7fa7a1e00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7a4065bf0 0x7fa7a411a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.368+0000 7fa799400700 1 -- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7a4065bf0 msgr2=0x7fa7a411a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.368+0000 7fa799400700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7a4065bf0 0x7fa7a411a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.368+0000 7fa799400700 1 -- 172.21.15.44:0/410224365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c005040 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.368+0000 7fa799400700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 0x7fa7a411a710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009180 tx=0x7fa78c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.369+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78c00e040 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.369+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa78c013420 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.369+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78c01c630 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.369+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a41143b0 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.369+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a4114970 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.370+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7a4114ea0 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.371+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa78c013590 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.372+0000 7fa79aa00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa7880692b0 0x7fa78806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.372+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa78c087ff0 con 0x7fa7a4101d40 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.372+0000 7fa79aa00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] conn(0x7fa78806ef80 0x7fa788071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.372+0000 7fa7a1e00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa7880692b0 0x7fa78806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.372+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 --> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa788071a40 con 0x7fa78806ef80 2024-04-16T11:38:27.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.372+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa78c08a530 con 0x7fa7a4101d40 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.373+0000 7fa7a2800700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] conn(0x7fa78806ef80 0x7fa788071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.373+0000 7fa7a1e00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa7880692b0 0x7fa78806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa794007fd0 tx=0x7fa794009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.374+0000 7fa7a2800700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] conn(0x7fa78806ef80 0x7fa788071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.376+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== osd.4 v2:172.21.15.45:6808/884231764 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa788071a40 con 0x7fa78806ef80 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.395+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 --> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa7a405f5f0 con 0x7fa78806ef80 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.396+0000 7fa79aa00700 1 -- 172.21.15.44:0/410224365 <== osd.4 v2:172.21.15.45:6808/884231764 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa7a405f5f0 con 0x7fa78806ef80 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] conn(0x7fa78806ef80 msgr2=0x7fa788071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] conn(0x7fa78806ef80 0x7fa788071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa7880692b0 msgr2=0x7fa78806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa7880692b0 0x7fa78806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa794007fd0 tx=0x7fa794009040 comp rx=0 tx=0).stop 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 msgr2=0x7fa7a411a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 0x7fa7a411a710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009180 tx=0x7fa78c009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 shutdown_connections 2024-04-16T11:38:27.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:6808/884231764,v1:172.21.15.45:6809/884231764] conn(0x7fa78806ef80 0x7fa788071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa7880692b0 0x7fa78806b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7a4065bf0 0x7fa7a411a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 --2- 172.21.15.44:0/410224365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7a4101d40 0x7fa7a411a710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 >> 172.21.15.44:0/410224365 conn(0x7fa7a4072bc0 msgr2=0x7fa7a40f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 shutdown_connections 2024-04-16T11:38:27.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.397+0000 7fa7a8a00700 1 -- 172.21.15.44:0/410224365 wait complete. 2024-04-16T11:38:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.702+0000 7fdade000700 1 -- 172.21.15.44:0/542818753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 msgr2=0x7fdad8109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.702+0000 7fdade000700 1 --2- 172.21.15.44:0/542818753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad8109890 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdac8004940 tx=0x7fdac8009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.703+0000 7fdade000700 1 -- 172.21.15.44:0/542818753 shutdown_connections 2024-04-16T11:38:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.703+0000 7fdade000700 1 --2- 172.21.15.44:0/542818753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad8103460 0x7fdad81038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.703+0000 7fdade000700 1 --2- 172.21.15.44:0/542818753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad8109890 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.703+0000 7fdade000700 1 -- 172.21.15.44:0/542818753 >> 172.21.15.44:0/542818753 conn(0x7fdad8072c30 msgr2=0x7fdad8075040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.703+0000 7fdade000700 1 -- 172.21.15.44:0/542818753 shutdown_connections 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.703+0000 7fdade000700 1 -- 172.21.15.44:0/542818753 wait complete. 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.704+0000 7fdade000700 1 Processor -- start 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.704+0000 7fdade000700 1 -- start start 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.704+0000 7fdade000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad8103460 0x7fdad811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.704+0000 7fdade000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad811a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.704+0000 7fdade000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad811ab90 con 0x7fdad81094c0 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.704+0000 7fdade000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad8114170 con 0x7fdad8103460 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad811a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad811a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45520/0 (socket says 172.21.15.44:45520) 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 -- 172.21.15.44:0/1409658751 learned_addr learned my addr 172.21.15.44:0/1409658751 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad7400700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad8103460 0x7fdad811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 -- 172.21.15.44:0/1409658751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad8103460 msgr2=0x7fdad811a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad8103460 0x7fdad811a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 -- 172.21.15.44:0/1409658751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8005040 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.705+0000 7fdad6a00700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad811a540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdac4002a50 tx=0x7fdac40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.706+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdac4007cc0 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.706+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdac400f040 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.706+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad8114450 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.706+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdac4009640 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.706+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad8114990 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.707+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdad8114ec0 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.708+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fdac40097a0 con 0x7fdad81094c0 2024-04-16T11:38:27.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.709+0000 7fdacf400700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fdabc0692a0 0x7fdabc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.709+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdac4086c20 con 0x7fdad81094c0 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.709+0000 7fdad7400700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fdabc0692a0 0x7fdabc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.709+0000 7fdacf400700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] conn(0x7fdabc06ef70 0x7fdabc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.709+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 --> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdabc071a30 con 0x7fdabc06ef70 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.709+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdac408b220 con 0x7fdad81094c0 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.710+0000 7fdad7e00700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] conn(0x7fdabc06ef70 0x7fdabc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.710+0000 7fdad7400700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fdabc0692a0 0x7fdabc06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdac800b3b0 tx=0x7fdac8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.710+0000 7fdad7e00700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] conn(0x7fdabc06ef70 0x7fdabc071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:27.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.711+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== osd.1 v2:172.21.15.44:6810/412128623 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdabc071a30 con 0x7fdabc06ef70 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.729+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 --> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdad81065d0 con 0x7fdabc06ef70 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.729+0000 7fdacf400700 1 -- 172.21.15.44:0/1409658751 <== osd.1 v2:172.21.15.44:6810/412128623 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdad81065d0 con 0x7fdabc06ef70 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.729+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] conn(0x7fdabc06ef70 msgr2=0x7fdabc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.729+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] conn(0x7fdabc06ef70 0x7fdabc071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fdabc0692a0 msgr2=0x7fdabc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fdabc0692a0 0x7fdabc06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdac800b3b0 tx=0x7fdac8009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 msgr2=0x7fdad811a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad811a540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdac4002a50 tx=0x7fdac40046b0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 shutdown_connections 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fdabc0692a0 0x7fdabc06b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:6810/412128623,v1:172.21.15.44:6811/412128623] conn(0x7fdabc06ef70 0x7fdabc071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad8103460 0x7fdad811a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 --2- 172.21.15.44:0/1409658751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdad81094c0 0x7fdad811a540 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:27.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 >> 172.21.15.44:0/1409658751 conn(0x7fdad8072c30 msgr2=0x7fdad8105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:27.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 shutdown_connections 2024-04-16T11:38:27.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:27.730+0000 7fdade000700 1 -- 172.21.15.44:0/1409658751 wait complete. 2024-04-16T11:38:28.641 INFO:teuthology.orchestra.run.smithi044.stdout:60129542181 2024-04-16T11:38:28.642 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-16T11:38:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:28 smithi044 ceph-mon[28731]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:29.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:28 smithi045 ceph-mon[36308]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.360+0000 7f39faa00700 1 -- 172.21.15.44:0/4228271577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 msgr2=0x7f39f410d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.360+0000 7f39faa00700 1 --2- 172.21.15.44:0/4228271577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f410d800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39e0004940 tx=0x7f39e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.360+0000 7f39faa00700 1 -- 172.21.15.44:0/4228271577 shutdown_connections 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.360+0000 7f39faa00700 1 --2- 172.21.15.44:0/4228271577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f410d800 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.360+0000 7f39faa00700 1 --2- 172.21.15.44:0/4228271577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39f40724f0 0x7f39f40728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.360+0000 7f39faa00700 1 -- 172.21.15.44:0/4228271577 >> 172.21.15.44:0/4228271577 conn(0x7f39f40fac00 msgr2=0x7f39f40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.361+0000 7f39faa00700 1 -- 172.21.15.44:0/4228271577 shutdown_connections 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.361+0000 7f39faa00700 1 -- 172.21.15.44:0/4228271577 wait complete. 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39faa00700 1 Processor -- start 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39faa00700 1 -- start start 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39faa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39f40724f0 0x7f39f4119ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39faa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f411a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39faa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f411aa70 con 0x7f39f4072e00 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39faa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f4114090 con 0x7f39f40724f0 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.362+0000 7f39f3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f411a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f411a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45528/0 (socket says 172.21.15.44:45528) 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3400700 1 -- 172.21.15.44:0/2394825636 learned_addr learned my addr 172.21.15.44:0/2394825636 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3e00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39f40724f0 0x7f39f4119ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3400700 1 -- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39f40724f0 msgr2=0x7f39f4119ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3400700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39f40724f0 0x7f39f4119ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3400700 1 -- 172.21.15.44:0/2394825636 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0005040 con 0x7f39f4072e00 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.363+0000 7f39f3400700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f411a420 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f39e0004890 tx=0x7f39e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.364+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e000e040 con 0x7f39f4072e00 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.364+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39e0009af0 con 0x7f39f4072e00 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.364+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f4114310 con 0x7f39f4072e00 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.364+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e001c630 con 0x7f39f4072e00 2024-04-16T11:38:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.364+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f41148d0 con 0x7f39f4072e00 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.365+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39f4114e40 con 0x7f39f4072e00 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.366+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f39e000bbc0 con 0x7f39f4072e00 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.367+0000 7f39ebe00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f39d80692a0 0x7f39d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.367+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f39e0088be0 con 0x7f39f4072e00 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.367+0000 7f39f3e00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f39d80692a0 0x7f39d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.367+0000 7f39ebe00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] conn(0x7f39d806ef70 0x7f39d8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.367+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 --> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39d8071a30 con 0x7f39d806ef70 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.367+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39e0088f20 con 0x7f39f4072e00 2024-04-16T11:38:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.368+0000 7f39f3e00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f39d80692a0 0x7f39d806b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f39e4004990 tx=0x7f39e4009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.368+0000 7f39f8a00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] conn(0x7f39d806ef70 0x7f39d8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.369+0000 7f39f8a00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] conn(0x7f39d806ef70 0x7f39d8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.371+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== osd.5 v2:172.21.15.45:6816/1526857616 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f39d8071a30 con 0x7f39d806ef70 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.388+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 --> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39f40fcbc0 con 0x7f39d806ef70 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.389+0000 7f39ebe00700 1 -- 172.21.15.44:0/2394825636 <== osd.5 v2:172.21.15.45:6816/1526857616 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f39f40fcbc0 con 0x7f39d806ef70 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] conn(0x7f39d806ef70 msgr2=0x7f39d8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] conn(0x7f39d806ef70 0x7f39d8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f39d80692a0 msgr2=0x7f39d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f39d80692a0 0x7f39d806b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f39e4004990 tx=0x7f39e4009ee0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 msgr2=0x7f39f411a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f411a420 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f39e0004890 tx=0x7f39e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 shutdown_connections 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.390+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:6816/1526857616,v1:172.21.15.45:6817/1526857616] conn(0x7f39d806ef70 0x7f39d8071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.391+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f39d80692a0 0x7f39d806b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.391+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39f40724f0 0x7f39f4119ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.391+0000 7f39faa00700 1 --2- 172.21.15.44:0/2394825636 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39f4072e00 0x7f39f411a420 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.391+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 >> 172.21.15.44:0/2394825636 conn(0x7f39f40fac00 msgr2=0x7f39f40fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.391+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 shutdown_connections 2024-04-16T11:38:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:29.391+0000 7f39faa00700 1 -- 172.21.15.44:0/2394825636 wait complete. 2024-04-16T11:38:30.775 INFO:teuthology.orchestra.run.smithi044.stdout:146028888081 2024-04-16T11:38:30.776 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-16T11:38:31.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:30 smithi045 ceph-mon[36308]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:30 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:31.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:30 smithi044 ceph-mon[28731]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:31.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:30 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:31.559 INFO:teuthology.orchestra.run.smithi044.stdout:167503724554 2024-04-16T11:38:31.559 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-16T11:38:31.684 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:32.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.483+0000 7fb6da400700 1 -- 172.21.15.44:0/808051495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fc2b0 msgr2=0x7fb6d4109010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.483+0000 7fb6da400700 1 --2- 172.21.15.44:0/808051495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4109010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0004970 tx=0x7fb6c0009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.484+0000 7fb6da400700 1 -- 172.21.15.44:0/808051495 shutdown_connections 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.484+0000 7fb6da400700 1 --2- 172.21.15.44:0/808051495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4109010 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.484+0000 7fb6da400700 1 --2- 172.21.15.44:0/808051495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d40fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.484+0000 7fb6da400700 1 -- 172.21.15.44:0/808051495 >> 172.21.15.44:0/808051495 conn(0x7fb6d40f7670 msgr2=0x7fb6d40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.484+0000 7fb6da400700 1 -- 172.21.15.44:0/808051495 shutdown_connections 2024-04-16T11:38:32.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.484+0000 7fb6da400700 1 -- 172.21.15.44:0/808051495 wait complete. 2024-04-16T11:38:32.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.485+0000 7fb6da400700 1 Processor -- start 2024-04-16T11:38:32.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.485+0000 7fb6da400700 1 -- start start 2024-04-16T11:38:32.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.485+0000 7fb6da400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d4115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.485+0000 7fb6da400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4116010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.485+0000 7fb6da400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d4116660 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.485+0000 7fb6da400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d410fa50 con 0x7fb6d40fc2b0 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d4115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d4115ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42090/0 (socket says 172.21.15.44:42090) 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 -- 172.21.15.44:0/3117344537 learned_addr learned my addr 172.21.15.44:0/3117344537 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d3e00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 -- 172.21.15.44:0/3117344537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fc2b0 msgr2=0x7fb6d4116010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4116010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 -- 172.21.15.44:0/3117344537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c0005040 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d3e00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.486+0000 7fb6d9000700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d4115ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800c8c0 tx=0x7fb6c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.487+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6c800a980 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.487+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6c800ce60 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.487+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d410fd30 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.487+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6c80115e0 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.487+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d41102f0 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.488+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6d41108d0 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.489+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb6c8012400 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.490+0000 7fb6d0c00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb6bc0692a0 0x7fb6bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.490+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb6c8087d00 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.490+0000 7fb6d3e00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb6bc0692a0 0x7fb6bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.490+0000 7fb6d0c00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] conn(0x7fb6bc06ef70 0x7fb6bc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.490+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 --> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6bc071a30 con 0x7fb6bc06ef70 2024-04-16T11:38:32.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.490+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6c8088040 con 0x7fb6d40fb9a0 2024-04-16T11:38:32.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.491+0000 7fb6d9a00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] conn(0x7fb6bc06ef70 0x7fb6bc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.491+0000 7fb6d3e00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb6bc0692a0 0x7fb6bc06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009fd0 tx=0x7fb6c0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.492+0000 7fb6d9a00700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] conn(0x7fb6bc06ef70 0x7fb6bc071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.493+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== osd.2 v2:172.21.15.44:6818/2239266596 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb6bc071a30 con 0x7fb6bc06ef70 2024-04-16T11:38:32.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.509+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 --> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb6d405e6c0 con 0x7fb6bc06ef70 2024-04-16T11:38:32.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.510+0000 7fb6d0c00700 1 -- 172.21.15.44:0/3117344537 <== osd.2 v2:172.21.15.44:6818/2239266596 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb6d405e6c0 con 0x7fb6bc06ef70 2024-04-16T11:38:32.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.510+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] conn(0x7fb6bc06ef70 msgr2=0x7fb6bc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.510+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] conn(0x7fb6bc06ef70 0x7fb6bc071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.510+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb6bc0692a0 msgr2=0x7fb6bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.510+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb6bc0692a0 0x7fb6bc06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009fd0 tx=0x7fb6c0009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:32.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 msgr2=0x7fb6d4115ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d4115ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800c8c0 tx=0x7fb6c80044c0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 shutdown_connections 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6818/2239266596,v1:172.21.15.44:6819/2239266596] conn(0x7fb6bc06ef70 0x7fb6bc071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb6bc0692a0 0x7fb6bc06b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6d40fb9a0 0x7fb6d4115ad0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 --2- 172.21.15.44:0/3117344537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6d40fc2b0 0x7fb6d4116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 >> 172.21.15.44:0/3117344537 conn(0x7fb6d40f7670 msgr2=0x7fb6d40f7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 shutdown_connections 2024-04-16T11:38:32.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.511+0000 7fb6da400700 1 -- 172.21.15.44:0/3117344537 wait complete. 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 -- 172.21.15.44:0/1466406012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109280 msgr2=0x7f8b10109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 --2- 172.21.15.44:0/1466406012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109280 0x7f8b10109650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8b00004940 tx=0x7f8b00009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 -- 172.21.15.44:0/1466406012 shutdown_connections 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 --2- 172.21.15.44:0/1466406012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b10109c20 0x7f8b10103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 --2- 172.21.15.44:0/1466406012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109280 0x7f8b10109650 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 -- 172.21.15.44:0/1466406012 >> 172.21.15.44:0/1466406012 conn(0x7f8b10072a60 msgr2=0x7f8b10074e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 -- 172.21.15.44:0/1466406012 shutdown_connections 2024-04-16T11:38:32.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.721+0000 7f8b0f200700 1 -- 172.21.15.44:0/1466406012 wait complete. 2024-04-16T11:38:32.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.722+0000 7f8b0f200700 1 Processor -- start 2024-04-16T11:38:32.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.722+0000 7f8b0f200700 1 -- start start 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0f200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b10109280 0x7f8b1011a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 0x7f8b1011a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1011aba0 con 0x7f8b10109c20 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0f200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10113f70 con 0x7f8b10109280 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 0x7f8b1011a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 0x7f8b1011a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42108/0 (socket says 172.21.15.44:42108) 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0d400700 1 -- 172.21.15.44:0/3409171632 learned_addr learned my addr 172.21.15.44:0/3409171632 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:32.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.723+0000 7f8b0de00700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b10109280 0x7f8b1011a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b0d400700 1 -- 172.21.15.44:0/3409171632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b10109280 msgr2=0x7f8b1011a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b0d400700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b10109280 0x7f8b1011a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b0d400700 1 -- 172.21.15.44:0/3409171632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b00005040 con 0x7f8b10109c20 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b0d400700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 0x7f8b1011a550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c890 tx=0x7f8afc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8afc00a7d0 con 0x7f8b10109c20 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b10114250 con 0x7f8b10109c20 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.724+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8afc00ae00 con 0x7f8b10109c20 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.725+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8afc0113a0 con 0x7f8b10109c20 2024-04-16T11:38:32.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.725+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b10114810 con 0x7f8b10109c20 2024-04-16T11:38:32.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.726+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8af0000fd0 con 0x7f8b10109c20 2024-04-16T11:38:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.727+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8afc00a930 con 0x7f8b10109c20 2024-04-16T11:38:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.728+0000 7f8b06000700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f8af40692a0 0x7f8af406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.728+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8afc086dc0 con 0x7f8b10109c20 2024-04-16T11:38:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.728+0000 7f8b0de00700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f8af40692a0 0x7f8af406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.728+0000 7f8b06000700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] conn(0x7f8af406ef70 0x7f8af4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.728+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 --> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8af4071a30 con 0x7f8af406ef70 2024-04-16T11:38:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.728+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8afc089430 con 0x7f8b10109c20 2024-04-16T11:38:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.729+0000 7f8b0e800700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] conn(0x7f8af406ef70 0x7f8af4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.729+0000 7f8b0de00700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f8af40692a0 0x7f8af406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000b020 tx=0x7f8b00009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.729+0000 7f8b0e800700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] conn(0x7f8af406ef70 0x7f8af4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.730+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== osd.0 v2:172.21.15.44:6802/1592689770 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8af4071a30 con 0x7f8af406ef70 2024-04-16T11:38:32.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.747+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 --> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8af0002d30 con 0x7f8af406ef70 2024-04-16T11:38:32.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b06000700 1 -- 172.21.15.44:0/3409171632 <== osd.0 v2:172.21.15.44:6802/1592689770 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8af0002d30 con 0x7f8af406ef70 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] conn(0x7f8af406ef70 msgr2=0x7f8af4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] conn(0x7f8af406ef70 0x7f8af4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f8af40692a0 msgr2=0x7f8af406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f8af40692a0 0x7f8af406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000b020 tx=0x7f8b00009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 msgr2=0x7f8b1011a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.748+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 0x7f8b1011a550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c890 tx=0x7f8afc0044c0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 shutdown_connections 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6802/1592689770,v1:172.21.15.44:6803/1592689770] conn(0x7f8af406ef70 0x7f8af4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f8af40692a0 0x7f8af406b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b10109280 0x7f8b1011a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 --2- 172.21.15.44:0/3409171632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b10109c20 0x7f8b1011a550 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 >> 172.21.15.44:0/3409171632 conn(0x7f8b10072a60 msgr2=0x7f8b10074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 shutdown_connections 2024-04-16T11:38:32.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:32.749+0000 7f8b0f200700 1 -- 172.21.15.44:0/3409171632 wait complete. 2024-04-16T11:38:33.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:32 smithi045 ceph-mon[36308]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:33.357 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:33.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:32 smithi044 ceph-mon[28731]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:33.993 INFO:teuthology.orchestra.run.smithi044.stdout:81604378654 2024-04-16T11:38:33.994 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-16T11:38:34.603 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:34.603 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:35.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:34 smithi045 ceph-mon[36308]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:35.454 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:34 smithi044 ceph-mon[28731]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:35.987 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:36.020 INFO:teuthology.orchestra.run.smithi044.stdout:38654705709 2024-04-16T11:38:36.021 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-16T11:38:36.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.655+0000 7f1ae0c00700 1 -- 172.21.15.44:0/4092888493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 msgr2=0x7f1adc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:36.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.655+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/4092888493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc0fce10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8004940 tx=0x7f1ac8009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:36.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.656+0000 7f1ae0c00700 1 -- 172.21.15.44:0/4092888493 shutdown_connections 2024-04-16T11:38:36.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.656+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/4092888493 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc0fce10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:36.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.656+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/4092888493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1adc102e00 0x7f1adc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:36.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.656+0000 7f1ae0c00700 1 -- 172.21.15.44:0/4092888493 >> 172.21.15.44:0/4092888493 conn(0x7f1adc0f8970 msgr2=0x7f1adc0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.656+0000 7f1ae0c00700 1 -- 172.21.15.44:0/4092888493 shutdown_connections 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.656+0000 7f1ae0c00700 1 -- 172.21.15.44:0/4092888493 wait complete. 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ae0c00700 1 Processor -- start 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ae0c00700 1 -- start start 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ae0c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1adc102e00 0x7f1adc115d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ae0c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc1162d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ae0c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc116920 con 0x7f1adc1037a0 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ae0c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc10fd40 con 0x7f1adc102e00 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ad9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc1162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ad9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc1162d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42130/0 (socket says 172.21.15.44:42130) 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.657+0000 7f1ad9400700 1 -- 172.21.15.44:0/2946184024 learned_addr learned my addr 172.21.15.44:0/2946184024 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.658+0000 7f1ad9e00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1adc102e00 0x7f1adc115d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.658+0000 7f1ad9400700 1 -- 172.21.15.44:0/2946184024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1adc102e00 msgr2=0x7f1adc115d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.658+0000 7f1ad9400700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1adc102e00 0x7f1adc115d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:36.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.658+0000 7f1ad9400700 1 -- 172.21.15.44:0/2946184024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac8005040 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.658+0000 7f1ad9400700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc1162d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009100 tx=0x7f1ac8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.658+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ac800e040 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.659+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac8013420 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.659+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1adc10ffc0 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.659+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ac801c630 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.659+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc110580 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.660+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc1018c0 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.661+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1ac8013590 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.662+0000 7f1ad2000700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1ac00692b0 0x7f1ac006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.662+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ac80880a0 con 0x7f1adc1037a0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.663+0000 7f1ad9e00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1ac00692b0 0x7f1ac006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.663+0000 7f1ad9e00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1ac00692b0 0x7f1ac006b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1acc007fd0 tx=0x7f1acc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:36.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.667+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1ac8058180 con 0x7f1adc1037a0 2024-04-16T11:38:36.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.886+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1adc1113e0 con 0x7f1adc1037a0 2024-04-16T11:38:36.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.887+0000 7f1ad2000700 1 -- 172.21.15.44:0/2946184024 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1ac8057d10 con 0x7f1adc1037a0 2024-04-16T11:38:36.886 INFO:teuthology.orchestra.run.smithi044.stdout:111669149719 2024-04-16T11:38:36.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1ac00692b0 msgr2=0x7f1ac006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1ac00692b0 0x7f1ac006b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1acc007fd0 tx=0x7f1acc009040 comp rx=0 tx=0).stop 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 msgr2=0x7f1adc1162d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc1162d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009100 tx=0x7f1ac8009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 shutdown_connections 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1ac00692b0 0x7f1ac006b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.890+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1adc102e00 0x7f1adc115d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.891+0000 7f1ae0c00700 1 --2- 172.21.15.44:0/2946184024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1adc1037a0 0x7f1adc1162d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:36.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.891+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 >> 172.21.15.44:0/2946184024 conn(0x7f1adc0f8970 msgr2=0x7f1adc102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:36.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.891+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 shutdown_connections 2024-04-16T11:38:36.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:36.891+0000 7f1ae0c00700 1 -- 172.21.15.44:0/2946184024 wait complete. 2024-04-16T11:38:37.214 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:36 smithi044 ceph-mon[28731]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:37.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:36 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2946184024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T11:38:37.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:36 smithi045 ceph-mon[36308]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:37.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:36 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2946184024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T11:38:39.147 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:39 smithi044 ceph-mon[28731]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:39 smithi045 ceph-mon[36308]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:40.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:40 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1702130992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T11:38:40.457 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:40 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1702130992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T11:38:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.898+0000 7f4981200700 1 -- 172.21.15.44:0/1070357665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c102e30 msgr2=0x7f497c103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.898+0000 7f4981200700 1 --2- 172.21.15.44:0/1070357665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c102e30 0x7f497c103200 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f496c004940 tx=0x7f496c009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.898+0000 7f4981200700 1 -- 172.21.15.44:0/1070357665 shutdown_connections 2024-04-16T11:38:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.898+0000 7f4981200700 1 --2- 172.21.15.44:0/1070357665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c1037d0 0x7f497c0fce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.898+0000 7f4981200700 1 --2- 172.21.15.44:0/1070357665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c102e30 0x7f497c103200 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.898+0000 7f4981200700 1 -- 172.21.15.44:0/1070357665 >> 172.21.15.44:0/1070357665 conn(0x7f497c0f8a20 msgr2=0x7f497c0fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.899+0000 7f4981200700 1 -- 172.21.15.44:0/1070357665 shutdown_connections 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.899+0000 7f4981200700 1 -- 172.21.15.44:0/1070357665 wait complete. 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.899+0000 7f4981200700 1 Processor -- start 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4981200700 1 -- start start 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4981200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c102e30 0x7f497c115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4981200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 0x7f497c1160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4981200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c116730 con 0x7f497c1037d0 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4981200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c10fd10 con 0x7f497c102e30 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4979400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 0x7f497c1160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4979400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 0x7f497c1160e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42148/0 (socket says 172.21.15.44:42148) 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.900+0000 7f4979400700 1 -- 172.21.15.44:0/1702130992 learned_addr learned my addr 172.21.15.44:0/1702130992 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4979e00700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c102e30 0x7f497c115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4979400700 1 -- 172.21.15.44:0/1702130992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c102e30 msgr2=0x7f497c115ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4979400700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c102e30 0x7f497c115ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4979400700 1 -- 172.21.15.44:0/1702130992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c005040 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4979400700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 0x7f497c1160e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f496800c800 tx=0x7f4968004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f496800a760 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c10fff0 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f496800ad90 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.901+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49680113a0 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.902+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c1105b0 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.903+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c106a30 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.904+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f4968012400 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.905+0000 7f4972000700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49600692a0 0x7f496006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.905+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4968086ea0 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.905+0000 7f4979e00700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49600692a0 0x7f496006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.906+0000 7f4979e00700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49600692a0 0x7f496006b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b3b0 tx=0x7f496c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:38.909+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4968056fa0 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.137+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f497c0fe580 con 0x7f497c1037d0 2024-04-16T11:38:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.137+0000 7f4972000700 1 -- 172.21.15.44:0/1702130992 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4968056b30 con 0x7f497c1037d0 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stdout:167503724556 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49600692a0 msgr2=0x7f496006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49600692a0 0x7f496006b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b3b0 tx=0x7f496c009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 msgr2=0x7f497c1160e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 0x7f497c1160e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f496800c800 tx=0x7f4968004470 comp rx=0 tx=0).stop 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 shutdown_connections 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49600692a0 0x7f496006b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f497c102e30 0x7f497c115ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:40.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 --2- 172.21.15.44:0/1702130992 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f497c1037d0 0x7f497c1160e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:40.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.140+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 >> 172.21.15.44:0/1702130992 conn(0x7f497c0f8a20 msgr2=0x7f497c107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:40.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.141+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 shutdown_connections 2024-04-16T11:38:40.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:39.141+0000 7f4981200700 1 -- 172.21.15.44:0/1702130992 wait complete. 2024-04-16T11:38:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:41 smithi044 ceph-mon[28731]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:41.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:41 smithi045 ceph-mon[36308]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:41.345 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-04-16T11:38:41.345 DEBUG:teuthology.parallel:result is None 2024-04-16T11:38:41.451 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149717 got 111669149719 for osd.3 2024-04-16T11:38:41.452 DEBUG:teuthology.parallel:result is None 2024-04-16T11:38:42.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 -- 172.21.15.44:0/1516410518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 msgr2=0x7ff7e8070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:42.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 --2- 172.21.15.44:0/1516410518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e8070360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004970 tx=0x7ff7dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:42.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 -- 172.21.15.44:0/1516410518 shutdown_connections 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 --2- 172.21.15.44:0/1516410518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 0x7ff7e8070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 --2- 172.21.15.44:0/1516410518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e8070360 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 -- 172.21.15.44:0/1516410518 >> 172.21.15.44:0/1516410518 conn(0x7ff7e80f89d0 msgr2=0x7ff7e80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 -- 172.21.15.44:0/1516410518 shutdown_connections 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.648+0000 7ff7efa00700 1 -- 172.21.15.44:0/1516410518 wait complete. 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.649+0000 7ff7efa00700 1 Processor -- start 2024-04-16T11:38:42.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.649+0000 7ff7efa00700 1 -- start start 2024-04-16T11:38:42.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7efa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 0x7ff7e811a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:42.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7efa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e811a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7efa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e811ad10 con 0x7ff7e8071f90 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7efa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e81140e0 con 0x7ff7e80708a0 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7e7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e811a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7e7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e811a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47148/0 (socket says 172.21.15.44:47148) 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7e7e00700 1 -- 172.21.15.44:0/3124272433 learned_addr learned my addr 172.21.15.44:0/3124272433 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.650+0000 7ff7ed000700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 0x7ff7e811a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e7e00700 1 -- 172.21.15.44:0/3124272433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 msgr2=0x7ff7e811a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e7e00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 0x7ff7e811a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e7e00700 1 -- 172.21.15.44:0/3124272433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc005040 con 0x7ff7e8071f90 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7ed000700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 0x7ff7e811a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e7e00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e811a6c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4002a50 tx=0x7ff7d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d4007cc0 con 0x7ff7e8071f90 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7d400f040 con 0x7ff7e8071f90 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e81143c0 con 0x7ff7e8071f90 2024-04-16T11:38:42.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d4009640 con 0x7ff7e8071f90 2024-04-16T11:38:42.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.651+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e8114900 con 0x7ff7e8071f90 2024-04-16T11:38:42.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.653+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e8071aa0 con 0x7ff7e8071f90 2024-04-16T11:38:42.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.655+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff7d40097a0 con 0x7ff7e8071f90 2024-04-16T11:38:42.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.656+0000 7ff7e4c00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7d00692b0 0x7ff7d006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:42.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.656+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff7d4086c20 con 0x7ff7e8071f90 2024-04-16T11:38:42.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.656+0000 7ff7ed000700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7d00692b0 0x7ff7d006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:42.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.657+0000 7ff7ed000700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7d00692b0 0x7ff7d006b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004ee0 tx=0x7ff7dc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:42.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.661+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff7d4056c80 con 0x7ff7e8071f90 2024-04-16T11:38:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.895+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff7e805f5f0 con 0x7ff7e8071f90 2024-04-16T11:38:42.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.895+0000 7ff7e4c00700 1 -- 172.21.15.44:0/3124272433 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff7d4056810 con 0x7ff7e8071f90 2024-04-16T11:38:42.893 INFO:teuthology.orchestra.run.smithi044.stdout:60129542184 2024-04-16T11:38:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7d00692b0 msgr2=0x7ff7d006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7d00692b0 0x7ff7d006b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004ee0 tx=0x7ff7dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 msgr2=0x7ff7e811a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:42.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e811a6c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4002a50 tx=0x7ff7d40046b0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 shutdown_connections 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7d00692b0 0x7ff7d006b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7e80708a0 0x7ff7e811a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 --2- 172.21.15.44:0/3124272433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7e8071f90 0x7ff7e811a6c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.898+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 >> 172.21.15.44:0/3124272433 conn(0x7ff7e80f89d0 msgr2=0x7ff7e80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.899+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 shutdown_connections 2024-04-16T11:38:42.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:42.899+0000 7ff7efa00700 1 -- 172.21.15.44:0/3124272433 wait complete. 2024-04-16T11:38:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:43 smithi045 ceph-mon[36308]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:43 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3124272433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T11:38:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:43 smithi044 ceph-mon[28731]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:43 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3124272433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 -- 172.21.15.44:0/2456981320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe834102e60 msgr2=0x7fe834103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 --2- 172.21.15.44:0/2456981320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe834102e60 0x7fe834103230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe824004940 tx=0x7fe824009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 -- 172.21.15.44:0/2456981320 shutdown_connections 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 --2- 172.21.15.44:0/2456981320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8340656b0 0x7fe834065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 --2- 172.21.15.44:0/2456981320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe834102e60 0x7fe834103230 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 -- 172.21.15.44:0/2456981320 >> 172.21.15.44:0/2456981320 conn(0x7fe834072bc0 msgr2=0x7fe834072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 -- 172.21.15.44:0/2456981320 shutdown_connections 2024-04-16T11:38:43.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.904+0000 7fe83a200700 1 -- 172.21.15.44:0/2456981320 wait complete. 2024-04-16T11:38:43.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.905+0000 7fe83a200700 1 Processor -- start 2024-04-16T11:38:43.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.905+0000 7fe83a200700 1 -- start start 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe83a200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 0x7fe83410b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe83a200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe834102e60 0x7fe83410bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe83a200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83410c360 con 0x7fe8340656b0 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe83a200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83410c4a0 con 0x7fe834102e60 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe833400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 0x7fe83410b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe833400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 0x7fe83410b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47172/0 (socket says 172.21.15.44:47172) 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe833400700 1 -- 172.21.15.44:0/511522779 learned_addr learned my addr 172.21.15.44:0/511522779 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:43.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.906+0000 7fe832a00700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe834102e60 0x7fe83410bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:43.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.907+0000 7fe833400700 1 -- 172.21.15.44:0/511522779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe834102e60 msgr2=0x7fe83410bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:43.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.907+0000 7fe833400700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe834102e60 0x7fe83410bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:43.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.907+0000 7fe833400700 1 -- 172.21.15.44:0/511522779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe824005040 con 0x7fe8340656b0 2024-04-16T11:38:43.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.907+0000 7fe833400700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 0x7fe83410b890 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe824009150 tx=0x7fe824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:43.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.908+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe82400e040 con 0x7fe8340656b0 2024-04-16T11:38:43.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.908+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83410c6f0 con 0x7fe8340656b0 2024-04-16T11:38:43.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.908+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe824013420 con 0x7fe8340656b0 2024-04-16T11:38:43.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.908+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe82401c610 con 0x7fe8340656b0 2024-04-16T11:38:43.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.908+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83411e680 con 0x7fe8340656b0 2024-04-16T11:38:43.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.910+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe834107320 con 0x7fe8340656b0 2024-04-16T11:38:43.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.911+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe82400baf0 con 0x7fe8340656b0 2024-04-16T11:38:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.911+0000 7fe82b400700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe8180692b0 0x7fe81806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.912+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe824088060 con 0x7fe8340656b0 2024-04-16T11:38:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.912+0000 7fe832a00700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe8180692b0 0x7fe81806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.912+0000 7fe832a00700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe8180692b0 0x7fe81806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe83410d1d0 tx=0x7fe82000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:43.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:43.917+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe8240546b0 con 0x7fe8340656b0 2024-04-16T11:38:44.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.136+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe83404bed0 con 0x7fe8340656b0 2024-04-16T11:38:44.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.137+0000 7fe82b400700 1 -- 172.21.15.44:0/511522779 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe824057cd0 con 0x7fe8340656b0 2024-04-16T11:38:44.134 INFO:teuthology.orchestra.run.smithi044.stdout:81604378656 2024-04-16T11:38:44.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.139+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe8180692b0 msgr2=0x7fe81806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:44.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.139+0000 7fe83a200700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe8180692b0 0x7fe81806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe83410d1d0 tx=0x7fe82000b040 comp rx=0 tx=0).stop 2024-04-16T11:38:44.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.139+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 msgr2=0x7fe83410b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:44.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.139+0000 7fe83a200700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 0x7fe83410b890 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe824009150 tx=0x7fe824009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:44.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.139+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 shutdown_connections 2024-04-16T11:38:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.139+0000 7fe83a200700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fe8180692b0 0x7fe81806b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.140+0000 7fe83a200700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8340656b0 0x7fe83410b890 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.140+0000 7fe83a200700 1 --2- 172.21.15.44:0/511522779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe834102e60 0x7fe83410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.140+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 >> 172.21.15.44:0/511522779 conn(0x7fe834072bc0 msgr2=0x7fe8340fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.140+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 shutdown_connections 2024-04-16T11:38:44.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:44.140+0000 7fe83a200700 1 -- 172.21.15.44:0/511522779 wait complete. 2024-04-16T11:38:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.018+0000 7f8493800700 1 -- 172.21.15.44:0/447601305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 msgr2=0x7f848c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.018+0000 7f8493800700 1 --2- 172.21.15.44:0/447601305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c1031f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8480004940 tx=0x7f8480009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.018+0000 7f8493800700 1 -- 172.21.15.44:0/447601305 shutdown_connections 2024-04-16T11:38:45.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.018+0000 7f8493800700 1 --2- 172.21.15.44:0/447601305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f848c1037c0 0x7f848c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.018+0000 7f8493800700 1 --2- 172.21.15.44:0/447601305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c1031f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.018+0000 7f8493800700 1 -- 172.21.15.44:0/447601305 >> 172.21.15.44:0/447601305 conn(0x7f848c0f89b0 msgr2=0x7f848c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:45.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.019+0000 7f8493800700 1 -- 172.21.15.44:0/447601305 shutdown_connections 2024-04-16T11:38:45.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.019+0000 7f8493800700 1 -- 172.21.15.44:0/447601305 wait complete. 2024-04-16T11:38:45.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.020+0000 7f8493800700 1 Processor -- start 2024-04-16T11:38:45.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.020+0000 7f8493800700 1 -- start start 2024-04-16T11:38:45.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.020+0000 7f8493800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:45.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.020+0000 7f8493800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f848c1037c0 0x7f848c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:45.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.020+0000 7f8493800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c072210 con 0x7f848c102e20 2024-04-16T11:38:45.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.020+0000 7f8493800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c070d40 con 0x7f848c1037c0 2024-04-16T11:38:45.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f8490e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f8490e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47178/0 (socket says 172.21.15.44:47178) 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f8490e00700 1 -- 172.21.15.44:0/416965157 learned_addr learned my addr 172.21.15.44:0/416965157 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f848be00700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f848c1037c0 0x7f848c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f8490e00700 1 -- 172.21.15.44:0/416965157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f848c1037c0 msgr2=0x7f848c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f8490e00700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f848c1037c0 0x7f848c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.021+0000 7f8490e00700 1 -- 172.21.15.44:0/416965157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480005040 con 0x7f848c102e20 2024-04-16T11:38:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.022+0000 7f8490e00700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c0702c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8480009180 tx=0x7f8480009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.022+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f848000e040 con 0x7f848c102e20 2024-04-16T11:38:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.023+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f848000bbc0 con 0x7f848c102e20 2024-04-16T11:38:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.023+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c070fc0 con 0x7f848c102e20 2024-04-16T11:38:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.023+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f848001c630 con 0x7f848c102e20 2024-04-16T11:38:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.023+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c11a110 con 0x7f848c102e20 2024-04-16T11:38:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.024+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c0052e0 con 0x7f848c102e20 2024-04-16T11:38:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.026+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f848001c790 con 0x7f848c102e20 2024-04-16T11:38:45.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.027+0000 7f8488c00700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84740692a0 0x7f847406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:45.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.027+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8480088250 con 0x7f848c102e20 2024-04-16T11:38:45.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.030+0000 7f848be00700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84740692a0 0x7f847406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:45.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.030+0000 7f848be00700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84740692a0 0x7f847406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f848c0719f0 tx=0x7f8478009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:45.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.032+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f84800548a0 con 0x7f848c102e20 2024-04-16T11:38:45.138 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:45 smithi044 ceph-mon[28731]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:45.138 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:45 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/511522779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T11:38:45.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.252+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f846c005590 con 0x7f848c102e20 2024-04-16T11:38:45.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.253+0000 7f8488c00700 1 -- 172.21.15.44:0/416965157 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8480054020 con 0x7f848c102e20 2024-04-16T11:38:45.250 INFO:teuthology.orchestra.run.smithi044.stdout:146028888085 2024-04-16T11:38:45.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84740692a0 msgr2=0x7f847406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:45.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84740692a0 0x7f847406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f848c0719f0 tx=0x7f8478009ee0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 msgr2=0x7f848c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:45.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c0702c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8480009180 tx=0x7f8480009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:45.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 shutdown_connections 2024-04-16T11:38:45.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84740692a0 0x7f847406b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f848c102e20 0x7f848c0702c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 --2- 172.21.15.44:0/416965157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f848c1037c0 0x7f848c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:45.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.255+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 >> 172.21.15.44:0/416965157 conn(0x7f848c0f89b0 msgr2=0x7f848c107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:45.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.256+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 shutdown_connections 2024-04-16T11:38:45.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:45.256+0000 7f8493800700 1 -- 172.21.15.44:0/416965157 wait complete. 2024-04-16T11:38:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:45 smithi045 ceph-mon[36308]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:45 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/511522779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T11:38:45.558 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542181 got 60129542184 for osd.1 2024-04-16T11:38:45.558 DEBUG:teuthology.parallel:result is None 2024-04-16T11:38:45.675 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:45.727 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378654 got 81604378656 for osd.2 2024-04-16T11:38:45.727 DEBUG:teuthology.parallel:result is None 2024-04-16T11:38:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:46 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/416965157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T11:38:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:46 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/416965157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T11:38:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:38:46.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888081 got 146028888085 for osd.4 2024-04-16T11:38:46.410 DEBUG:teuthology.parallel:result is None 2024-04-16T11:38:47.214 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:47 smithi044 ceph-mon[28731]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:47 smithi045 ceph-mon[36308]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 -- 172.21.15.44:0/1709059961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 msgr2=0x7fa3640fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 --2- 172.21.15.44:0/1709059961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa3640fee90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa350004940 tx=0x7fa350009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 -- 172.21.15.44:0/1709059961 shutdown_connections 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 --2- 172.21.15.44:0/1709059961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa3640fee90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 --2- 172.21.15.44:0/1709059961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa364104e80 0x7fa364105250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 -- 172.21.15.44:0/1709059961 >> 172.21.15.44:0/1709059961 conn(0x7fa3640fab10 msgr2=0x7fa3640fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:48.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 -- 172.21.15.44:0/1709059961 shutdown_connections 2024-04-16T11:38:48.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.299+0000 7fa363e00700 1 -- 172.21.15.44:0/1709059961 wait complete. 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.300+0000 7fa363e00700 1 Processor -- start 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa363e00700 1 -- start start 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa363e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa364104e80 0x7fa364115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa363e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa364116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa363e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3641167a0 con 0x7fa364105820 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa363e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36410fb90 con 0x7fa364104e80 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa362000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa364116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa362000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa364116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47200/0 (socket says 172.21.15.44:47200) 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.301+0000 7fa362000700 1 -- 172.21.15.44:0/2513366285 learned_addr learned my addr 172.21.15.44:0/2513366285 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:48.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.302+0000 7fa362a00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa364104e80 0x7fa364115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:48.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.302+0000 7fa362000700 1 -- 172.21.15.44:0/2513366285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa364104e80 msgr2=0x7fa364115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:48.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.302+0000 7fa362000700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa364104e80 0x7fa364115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.302+0000 7fa362000700 1 -- 172.21.15.44:0/2513366285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350005040 con 0x7fa364105820 2024-04-16T11:38:48.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.302+0000 7fa362000700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa364116150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa350009100 tx=0x7fa350009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:48.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.303+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35000e040 con 0x7fa364105820 2024-04-16T11:38:48.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.303+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa350013420 con 0x7fa364105820 2024-04-16T11:38:48.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.303+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35001c630 con 0x7fa364105820 2024-04-16T11:38:48.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.303+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36410fe10 con 0x7fa364105820 2024-04-16T11:38:48.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.303+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3641103d0 con 0x7fa364105820 2024-04-16T11:38:48.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.304+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3641005d0 con 0x7fa364105820 2024-04-16T11:38:48.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.305+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa350013590 con 0x7fa364105820 2024-04-16T11:38:48.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.306+0000 7fa35aa00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa3480692a0 0x7fa34806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:48.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.306+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa350088170 con 0x7fa364105820 2024-04-16T11:38:48.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.306+0000 7fa362a00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa3480692a0 0x7fa34806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:48.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.307+0000 7fa362a00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa3480692a0 0x7fa34806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa354004990 tx=0x7fa354009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:48.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.311+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa3500581d0 con 0x7fa364105820 2024-04-16T11:38:48.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.528+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa3640fe610 con 0x7fa364105820 2024-04-16T11:38:48.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.529+0000 7fa35aa00700 1 -- 172.21.15.44:0/2513366285 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa350057d60 con 0x7fa364105820 2024-04-16T11:38:48.526 INFO:teuthology.orchestra.run.smithi044.stdout:38654705712 2024-04-16T11:38:48.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.531+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa3480692a0 msgr2=0x7fa34806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:48.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.531+0000 7fa363e00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa3480692a0 0x7fa34806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa354004990 tx=0x7fa354009ee0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.531+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 msgr2=0x7fa364116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:48.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.531+0000 7fa363e00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa364116150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa350009100 tx=0x7fa350009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 shutdown_connections 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa3480692a0 0x7fa34806b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa364104e80 0x7fa364115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 --2- 172.21.15.44:0/2513366285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa364105820 0x7fa364116150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 >> 172.21.15.44:0/2513366285 conn(0x7fa3640fab10 msgr2=0x7fa3640fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 shutdown_connections 2024-04-16T11:38:48.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:48.532+0000 7fa363e00700 1 -- 172.21.15.44:0/2513366285 wait complete. 2024-04-16T11:38:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:49 smithi044 ceph-mon[28731]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:49 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2513366285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T11:38:49.288 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-04-16T11:38:49.288 DEBUG:teuthology.parallel:result is None 2024-04-16T11:38:49.289 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-16T11:38:49.289 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T11:38:49.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:49 smithi045 ceph-mon[36308]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:49.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:49 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2513366285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T11:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:51.308 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:51 smithi044 ceph-mon[28731]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:51.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:51 smithi045 ceph-mon[36308]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.815+0000 7ffa6d000700 1 -- 172.21.15.44:0/795084698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 msgr2=0x7ffa681053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.815+0000 7ffa6d000700 1 --2- 172.21.15.44:0/795084698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa681053e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa58004940 tx=0x7ffa58009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.816+0000 7ffa6d000700 1 -- 172.21.15.44:0/795084698 shutdown_connections 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.816+0000 7ffa6d000700 1 --2- 172.21.15.44:0/795084698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 0x7ffa680ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.816+0000 7ffa6d000700 1 --2- 172.21.15.44:0/795084698 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa681053e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.816+0000 7ffa6d000700 1 -- 172.21.15.44:0/795084698 >> 172.21.15.44:0/795084698 conn(0x7ffa680faba0 msgr2=0x7ffa680fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.816+0000 7ffa6d000700 1 -- 172.21.15.44:0/795084698 shutdown_connections 2024-04-16T11:38:51.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.816+0000 7ffa6d000700 1 -- 172.21.15.44:0/795084698 wait complete. 2024-04-16T11:38:51.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.817+0000 7ffa6d000700 1 Processor -- start 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.817+0000 7ffa6d000700 1 -- start start 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa6d000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa68115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa6d000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 0x7ffa68116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa6d000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681168d0 con 0x7ffa68105010 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa6d000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6810fce0 con 0x7ffa681059b0 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa65e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa68115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa65e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa68115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33246/0 (socket says 172.21.15.44:33246) 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.818+0000 7ffa65e00700 1 -- 172.21.15.44:0/4066088594 learned_addr learned my addr 172.21.15.44:0/4066088594 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa5fe00700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 0x7ffa68116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa65e00700 1 -- 172.21.15.44:0/4066088594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 msgr2=0x7ffa68116280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa65e00700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 0x7ffa68116280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:51.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa65e00700 1 -- 172.21.15.44:0/4066088594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa58005040 con 0x7ffa68105010 2024-04-16T11:38:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa5fe00700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 0x7ffa68116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:38:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa65e00700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa68115d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009150 tx=0x7ffa58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5800e040 con 0x7ffa68105010 2024-04-16T11:38:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa58013420 con 0x7ffa68105010 2024-04-16T11:38:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6810ff60 con 0x7ffa68105010 2024-04-16T11:38:51.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.819+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5801c630 con 0x7ffa68105010 2024-04-16T11:38:51.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.820+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa68110520 con 0x7ffa68105010 2024-04-16T11:38:51.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.821+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa68100760 con 0x7ffa68105010 2024-04-16T11:38:51.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.822+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ffa58013590 con 0x7ffa68105010 2024-04-16T11:38:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.822+0000 7ffa5e000700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ffa4c0692b0 0x7ffa4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.823+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ffa580880b0 con 0x7ffa68105010 2024-04-16T11:38:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.823+0000 7ffa5fe00700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ffa4c0692b0 0x7ffa4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:51.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.824+0000 7ffa5fe00700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ffa4c0692b0 0x7ffa4c06b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009fd0 tx=0x7ffa5000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:51.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:51.827+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ffa58054700 con 0x7ffa68105010 2024-04-16T11:38:52.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.043+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa6804bf40 con 0x7ffa4c0692b0 2024-04-16T11:38:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.047+0000 7ffa5e000700 1 -- 172.21.15.44:0/4066088594 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27643 (secure 0 0 0) 0x7ffa6804bf40 con 0x7ffa4c0692b0 2024-04-16T11:38:52.045 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ffa4c0692b0 msgr2=0x7ffa4c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ffa4c0692b0 0x7ffa4c06b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009fd0 tx=0x7ffa5000b040 comp rx=0 tx=0).stop 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 msgr2=0x7ffa68115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa68115d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009150 tx=0x7ffa58009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 shutdown_connections 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ffa4c0692b0 0x7ffa4c06b760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa68105010 0x7ffa68115d40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 --2- 172.21.15.44:0/4066088594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa681059b0 0x7ffa68116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.051+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 >> 172.21.15.44:0/4066088594 conn(0x7ffa680faba0 msgr2=0x7ffa680fbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.052+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 shutdown_connections 2024-04-16T11:38:52.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:52.052+0000 7ffa6d000700 1 -- 172.21.15.44:0/4066088594 wait complete. 2024-04-16T11:38:52.050 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-16T11:38:52.842 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-16T11:38:51.928548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788028,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682884,"statfs":{"total":575970213888,"available":574139273216,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007610"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-16T11:37:50.014389+0000","last_change":"2024-04-16T11:37:17.068929+0000","last_active":"2024-04-16T11:37:50.014389+0000","last_peered":"2024-04-16T11:37:50.014389+0000","last_clean":"2024-04-16T11:37:50.014389+0000","last_became_active":"2024-04-16T11:37:17.067610+0000","last_became_peered":"2024-04-16T11:37:17.067610+0000","last_unstale":"2024-04-16T11:37:50.014389+0000","last_undegraded":"2024-04-16T11:37:50.014389+0000","last_fullsized":"2024-04-16T11:37:50.014389+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T11:36:12.588707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T11:36:12.588707+0000","last_clean_scrub_stamp":"2024-04-16T11:36:12.588707+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T14:32:14.508571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]}]},{"osd":4,"up_from":34,"seq":146028888086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":3.1419999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":1.3919999999999999}]},{"osd":1,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":1.236},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":1.329}]},{"osd":2,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":1.085},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.5740000000000001}]},{"osd":3,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.649},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.5509999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0329999999999999,"15min":1.018},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.208},{"interface":"front","average":{"1min":1.085,"5min":0.97199999999999998,"15min":0.95299999999999996},"min":{"1min":0.67600000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.3959999999999999,"5min":1.542,"15min":1.542},"last":1.2290000000000001}]},{"osd":1,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.004,"15min":0.999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5740000000000001,"5min":1.665,"15min":1.665},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.006,"15min":0.99399999999999999},"min":{"1min":0.69999999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.1679999999999999}]},{"osd":2,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.80500000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":1.024},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.173,"15min":1.163},"min":{"1min":0.86599999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.538,"5min":1.593,"15min":1.593},"last":1.095}]},{"osd":4,"last update":"Tue Apr 16 11:38:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.369}]},{"osd":5,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.3220000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378657,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297708,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.81299999999999994,"15min":0.80000000000000004},"min":{"1min":0.55700000000000005,"5min":0.433,"15min":0.433},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.03},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.86799999999999999,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.92200000000000004}]},{"osd":1,"last update":"Tue Apr 16 11:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.92000000000000004,"15min":0.91100000000000003},"min":{"1min":0.61699999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.97299999999999998,"15min":0.96699999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.482,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.0629999999999999}]},{"osd":3,"last update":"Tue Apr 16 11:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":1.268},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.1739999999999999}]},{"osd":4,"last update":"Tue Apr 16 11:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.325},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":1.1970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 11:38:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.82699999999999996,"15min":0.81899999999999995},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.879},{"interface":"front","average":{"1min":1.032,"5min":0.90600000000000003,"15min":0.878},"min":{"1min":0.60699999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.83599999999999997}]},{"osd":2,"last update":"Tue Apr 16 11:38:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.98799999999999999,"15min":0.97199999999999998},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.173,"5min":1.0129999999999999,"15min":0.98699999999999999},"min":{"1min":0.83499999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Apr 16 11:38:47 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1279999999999999,"15min":1.1040000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.266,"15min":1.2450000000000001},"min":{"1min":0.98799999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0139999999999998,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.4390000000000001}]},{"osd":4,"last update":"Tue Apr 16 11:38:18 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.292,"15min":1.292},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.075},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":1.1699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:40 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.79700000000000004,"15min":0.75600000000000001},"min":{"1min":0.42399999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8919999999999999,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.85899999999999999,"15min":0.82699999999999996},"min":{"1min":0.63600000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.425,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Tue Apr 16 11:38:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.97399999999999998,"15min":0.97599999999999998},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.54,"5min":1.897,"15min":1.897},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1180000000000001,"15min":1.1279999999999999},"min":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.3069999999999999,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":1.077}]},{"osd":3,"last update":"Tue Apr 16 11:38:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.1830000000000001,"15min":1.177},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.218},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1639999999999999,"15min":1.159},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.98599999999999999}]},{"osd":4,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.93500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T11:38:52.843 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T11:38:53.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:53 smithi045 ceph-mon[36308]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:53.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:53 smithi045 ceph-mon[36308]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T11:38:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:53.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:53 smithi044 ceph-mon[28731]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:53.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:53 smithi044 ceph-mon[28731]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T11:38:55.323 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:55 smithi044 ceph-mon[28731]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:55.323 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:55 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:38:55.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:55 smithi045 ceph-mon[36308]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:55 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:38:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 -- 172.21.15.44:0/914849059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070300 msgr2=0x7faec80706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 --2- 172.21.15.44:0/914849059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070300 0x7faec80706d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004970 tx=0x7faeb8009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 -- 172.21.15.44:0/914849059 shutdown_connections 2024-04-16T11:38:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 --2- 172.21.15.44:0/914849059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070c10 0x7faec8108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 --2- 172.21.15.44:0/914849059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070300 0x7faec80706d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 -- 172.21.15.44:0/914849059 >> 172.21.15.44:0/914849059 conn(0x7faec80f87d0 msgr2=0x7faec80fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 -- 172.21.15.44:0/914849059 shutdown_connections 2024-04-16T11:38:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.493+0000 7faec7200700 1 -- 172.21.15.44:0/914849059 wait complete. 2024-04-16T11:38:55.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.494+0000 7faec7200700 1 Processor -- start 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.495+0000 7faec7200700 1 -- start start 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.495+0000 7faec7200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070300 0x7faec8115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.495+0000 7faec7200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 0x7faec81160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.495+0000 7faec7200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec8116710 con 0x7faec8070c10 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.495+0000 7faec7200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec810fb20 con 0x7faec8070300 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.495+0000 7faec5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 0x7faec81160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 0x7faec81160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33278/0 (socket says 172.21.15.44:33278) 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5400700 1 -- 172.21.15.44:0/349285592 learned_addr learned my addr 172.21.15.44:0/349285592 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5e00700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070300 0x7faec8115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5400700 1 -- 172.21.15.44:0/349285592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070300 msgr2=0x7faec8115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5400700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070300 0x7faec8115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5400700 1 -- 172.21.15.44:0/349285592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8005040 con 0x7faec8070c10 2024-04-16T11:38:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.496+0000 7faec5e00700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070300 0x7faec8115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T11:38:55.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.497+0000 7faec5400700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 0x7faec81160c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faeb400c8c0 tx=0x7faeb40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.498+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb400a760 con 0x7faec8070c10 2024-04-16T11:38:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.498+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faeb400ad90 con 0x7faec8070c10 2024-04-16T11:38:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.498+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec810fe00 con 0x7faec8070c10 2024-04-16T11:38:55.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.498+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb40113a0 con 0x7faec8070c10 2024-04-16T11:38:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.498+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec8110340 con 0x7faec8070c10 2024-04-16T11:38:55.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.499+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec8071a40 con 0x7faec8070c10 2024-04-16T11:38:55.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.501+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7faeb4012400 con 0x7faec8070c10 2024-04-16T11:38:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.502+0000 7faebe000700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faeac0692a0 0x7faeac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.503+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faeb4087a90 con 0x7faec8070c10 2024-04-16T11:38:55.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.503+0000 7faec5e00700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faeac0692a0 0x7faeac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:55.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.504+0000 7faec5e00700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faeac0692a0 0x7faeac06b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004f50 tx=0x7faeb8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:55.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.508+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7faeb40540e0 con 0x7faec8070c10 2024-04-16T11:38:55.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.723+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faec810bf50 con 0x7faeac0692a0 2024-04-16T11:38:55.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.727+0000 7faebe000700 1 -- 172.21.15.44:0/349285592 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28364 (secure 0 0 0) 0x7faec810bf50 con 0x7faeac0692a0 2024-04-16T11:38:55.725 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faeac0692a0 msgr2=0x7faeac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faeac0692a0 0x7faeac06b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004f50 tx=0x7faeb8009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 msgr2=0x7faec81160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 0x7faec81160c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faeb400c8c0 tx=0x7faeb40044c0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 shutdown_connections 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faeac0692a0 0x7faeac06b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec8070300 0x7faec8115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 --2- 172.21.15.44:0/349285592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faec8070c10 0x7faec81160c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.730+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 >> 172.21.15.44:0/349285592 conn(0x7faec80f87d0 msgr2=0x7faec8105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:55.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.731+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 shutdown_connections 2024-04-16T11:38:55.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:55.731+0000 7faec7200700 1 -- 172.21.15.44:0/349285592 wait complete. 2024-04-16T11:38:55.729 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-16T11:38:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:38:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:38:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:38:56.419 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:38:56.419 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:38:56.419 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:38:56.474 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-16T11:38:53.929166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788028,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682884,"statfs":{"total":575970213888,"available":574139273216,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007628"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-16T11:37:50.014389+0000","last_change":"2024-04-16T11:37:17.068929+0000","last_active":"2024-04-16T11:37:50.014389+0000","last_peered":"2024-04-16T11:37:50.014389+0000","last_clean":"2024-04-16T11:37:50.014389+0000","last_became_active":"2024-04-16T11:37:17.067610+0000","last_became_peered":"2024-04-16T11:37:17.067610+0000","last_unstale":"2024-04-16T11:37:50.014389+0000","last_undegraded":"2024-04-16T11:37:50.014389+0000","last_fullsized":"2024-04-16T11:37:50.014389+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T11:36:12.588707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T11:36:12.588707+0000","last_clean_scrub_stamp":"2024-04-16T11:36:12.588707+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T14:32:14.508571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]}]},{"osd":4,"up_from":34,"seq":146028888087,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":3.1419999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":1.0209999999999999}]},{"osd":1,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.5779999999999998,"5min":2.5779999999999998,"15min":2.5779999999999998},"last":1.091},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":1.2529999999999999}]},{"osd":2,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":1.054},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.208}]},{"osd":3,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.95299999999999996}]},{"osd":5,"last update":"Tue Apr 16 11:38:51 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.133}]}]},{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0329999999999999,"15min":1.018},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.208},{"interface":"front","average":{"1min":1.085,"5min":0.97199999999999998,"15min":0.95299999999999996},"min":{"1min":0.67600000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.3959999999999999,"5min":1.542,"15min":1.542},"last":1.2290000000000001}]},{"osd":1,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.004,"15min":0.999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5740000000000001,"5min":1.665,"15min":1.665},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.006,"15min":0.99399999999999999},"min":{"1min":0.69999999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.1679999999999999}]},{"osd":2,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.80500000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":1.024},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.173,"15min":1.163},"min":{"1min":0.86599999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.538,"5min":1.593,"15min":1.593},"last":1.095}]},{"osd":4,"last update":"Tue Apr 16 11:38:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.369}]},{"osd":5,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.3220000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297708,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.81299999999999994,"15min":0.80000000000000004},"min":{"1min":0.55700000000000005,"5min":0.433,"15min":0.433},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.86799999999999999,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.075}]},{"osd":1,"last update":"Tue Apr 16 11:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.92000000000000004,"15min":0.91100000000000003},"min":{"1min":0.61699999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.008},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.97299999999999998,"15min":0.96699999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.482,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.3520000000000001}]},{"osd":3,"last update":"Tue Apr 16 11:38:47 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.1819999999999999,"15min":1.1639999999999999},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.599},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1279999999999999,"15min":1.1180000000000001},"min":{"1min":0.91200000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":1.268}]},{"osd":4,"last update":"Tue Apr 16 11:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":1.522}]},{"osd":5,"last update":"Tue Apr 16 11:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.6639999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 11:38:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.82699999999999996,"15min":0.81899999999999995},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.879},{"interface":"front","average":{"1min":1.032,"5min":0.90600000000000003,"15min":0.878},"min":{"1min":0.60699999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.83599999999999997}]},{"osd":2,"last update":"Tue Apr 16 11:38:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.98799999999999999,"15min":0.97199999999999998},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.173,"5min":1.0129999999999999,"15min":0.98699999999999999},"min":{"1min":0.83499999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Apr 16 11:38:47 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1279999999999999,"15min":1.1040000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.266,"15min":1.2450000000000001},"min":{"1min":0.98799999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0139999999999998,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.4390000000000001}]},{"osd":4,"last update":"Tue Apr 16 11:38:18 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.292,"15min":1.292},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.075},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":1.1699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 11:38:40 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.79700000000000004,"15min":0.75600000000000001},"min":{"1min":0.42399999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8919999999999999,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.85899999999999999,"15min":0.82699999999999996},"min":{"1min":0.63600000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.425,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Tue Apr 16 11:38:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.97399999999999998,"15min":0.97599999999999998},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.54,"5min":1.897,"15min":1.897},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1180000000000001,"15min":1.1279999999999999},"min":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.3069999999999999,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":1.077}]},{"osd":3,"last update":"Tue Apr 16 11:38:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.1830000000000001,"15min":1.177},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.218},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1639999999999999,"15min":1.159},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.98599999999999999}]},{"osd":4,"last update":"Tue Apr 16 11:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.93500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T11:38:56.475 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-16T11:38:56.475 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T11:38:56.475 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-16T11:38:56.475 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-16T11:38:56.999 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:38:57.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:57 smithi045 ceph-mon[36308]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T11:38:57.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:57 smithi045 ceph-mon[36308]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:57.367 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:57 smithi044 ceph-mon[28731]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T11:38:57.367 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:57 smithi044 ceph-mon[28731]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:59.132 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:38:59 smithi045 ceph-mon[36308]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:59.173 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:38:59 smithi044 ceph-mon[28731]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:38:59.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 -- 172.21.15.44:0/2464601787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d8105010 msgr2=0x7f31d81053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 --2- 172.21.15.44:0/2464601787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d8105010 0x7f31d81053e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f31cc004940 tx=0x7f31cc009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 -- 172.21.15.44:0/2464601787 shutdown_connections 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 --2- 172.21.15.44:0/2464601787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d81059b0 0x7f31d80ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 --2- 172.21.15.44:0/2464601787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d8105010 0x7f31d81053e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 -- 172.21.15.44:0/2464601787 >> 172.21.15.44:0/2464601787 conn(0x7f31d80faba0 msgr2=0x7f31d80fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 -- 172.21.15.44:0/2464601787 shutdown_connections 2024-04-16T11:38:59.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.345+0000 7f31df400700 1 -- 172.21.15.44:0/2464601787 wait complete. 2024-04-16T11:38:59.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.346+0000 7f31df400700 1 Processor -- start 2024-04-16T11:38:59.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.346+0000 7f31df400700 1 -- start start 2024-04-16T11:38:59.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31df400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d8105010 0x7f31d8115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:59.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31df400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 0x7f31d81160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:59.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31df400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8116710 con 0x7f31d81059b0 2024-04-16T11:38:59.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31df400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d810fcf0 con 0x7f31d8105010 2024-04-16T11:38:59.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 0x7f31d81160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:59.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 0x7f31d81160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33304/0 (socket says 172.21.15.44:33304) 2024-04-16T11:38:59.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31d7e00700 1 -- 172.21.15.44:0/2178643253 learned_addr learned my addr 172.21.15.44:0/2178643253 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:38:59.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.347+0000 7f31dca00700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d8105010 0x7f31d8115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d7e00700 1 -- 172.21.15.44:0/2178643253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d8105010 msgr2=0x7f31d8115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d7e00700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d8105010 0x7f31d8115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d7e00700 1 -- 172.21.15.44:0/2178643253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc005040 con 0x7f31d81059b0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d7e00700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 0x7f31d81160c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31c400c800 tx=0x7f31c4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31c400a760 con 0x7f31d81059b0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31c400ad90 con 0x7f31d81059b0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31c40113a0 con 0x7f31d81059b0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.348+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d810ffd0 con 0x7f31d81059b0 2024-04-16T11:38:59.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.349+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d8110590 con 0x7f31d81059b0 2024-04-16T11:38:59.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.350+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d8100760 con 0x7f31d81059b0 2024-04-16T11:38:59.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.351+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f31c4012400 con 0x7f31d81059b0 2024-04-16T11:38:59.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.351+0000 7f31d4c00700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f31c00692a0 0x7f31c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:38:59.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.352+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f31c4086ea0 con 0x7f31d81059b0 2024-04-16T11:38:59.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.352+0000 7f31dca00700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f31c00692a0 0x7f31c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:38:59.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.353+0000 7f31dca00700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f31c00692a0 0x7f31c006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b3b0 tx=0x7f31cc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:38:59.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.356+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f31c4056f80 con 0x7f31d81059b0 2024-04-16T11:38:59.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.632+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31d804c710 con 0x7f31d81059b0 2024-04-16T11:38:59.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.632+0000 7f31d4c00700 1 -- 172.21.15.44:0/2178643253 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f31c4056b10 con 0x7f31d81059b0 2024-04-16T11:38:59.630 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:38:59.630 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T11:38:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f31c00692a0 msgr2=0x7f31c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f31c00692a0 0x7f31c006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b3b0 tx=0x7f31cc009e30 comp rx=0 tx=0).stop 2024-04-16T11:38:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 msgr2=0x7f31d81160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:38:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 0x7f31d81160c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31c400c800 tx=0x7f31c4004470 comp rx=0 tx=0).stop 2024-04-16T11:38:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 shutdown_connections 2024-04-16T11:38:59.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f31c00692a0 0x7f31c006b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:59.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31d8105010 0x7f31d8115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:59.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 --2- 172.21.15.44:0/2178643253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f31d81059b0 0x7f31d81160c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:38:59.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.635+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 >> 172.21.15.44:0/2178643253 conn(0x7f31d80faba0 msgr2=0x7f31d80fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:38:59.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.636+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 shutdown_connections 2024-04-16T11:38:59.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:38:59.636+0000 7f31df400700 1 -- 172.21.15.44:0/2178643253 wait complete. 2024-04-16T11:39:00.315 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:00 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2178643253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T11:39:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:00 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2178643253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T11:39:00.524 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-16T11:39:00.524 INFO:tasks.cephadm:Setup complete, yielding 2024-04-16T11:39:00.525 INFO:teuthology.run_tasks:Running task print... 2024-04-16T11:39:00.534 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-16T11:39:00.534 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:00.542 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:00.542 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-16T11:39:01.062 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:01 smithi045 ceph-mon[36308]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:01.375 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:01 smithi044 ceph-mon[28731]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:01.375 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:03.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 -- 172.21.15.44:0/3962740774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fce80 msgr2=0x7fece80fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 --2- 172.21.15.44:0/3962740774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fce80 0x7fece80fd290 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fecd8004940 tx=0x7fecd8009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 -- 172.21.15.44:0/3962740774 shutdown_connections 2024-04-16T11:39:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 --2- 172.21.15.44:0/3962740774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece80fdff0 0x7fece80fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 --2- 172.21.15.44:0/3962740774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fce80 0x7fece80fd290 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 -- 172.21.15.44:0/3962740774 >> 172.21.15.44:0/3962740774 conn(0x7fece80f8540 msgr2=0x7fece80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:03.367 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:03 smithi044 ceph-mon[28731]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 -- 172.21.15.44:0/3962740774 shutdown_connections 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.368+0000 7feced800700 1 -- 172.21.15.44:0/3962740774 wait complete. 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.369+0000 7feced800700 1 Processor -- start 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.369+0000 7feced800700 1 -- start start 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7feced800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece80fce80 0x7fece810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7feced800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 0x7fece81100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7feced800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece81106c0 con 0x7fece80fdff0 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7feced800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece8110830 con 0x7fece80fce80 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fecde000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 0x7fece81100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fecde000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 0x7fece81100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49912/0 (socket says 172.21.15.44:49912) 2024-04-16T11:39:03.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fecde000700 1 -- 172.21.15.44:0/1317421670 learned_addr learned my addr 172.21.15.44:0/1317421670 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:03.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fece6a00700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece80fce80 0x7fece810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:03.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fecde000700 1 -- 172.21.15.44:0/1317421670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece80fce80 msgr2=0x7fece810fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:03.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fecde000700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece80fce80 0x7fece810fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.370+0000 7fecde000700 1 -- 172.21.15.44:0/1317421670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd8005040 con 0x7fece80fdff0 2024-04-16T11:39:03.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.371+0000 7fecde000700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 0x7fece81100c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fecd0002a50 tx=0x7fecd00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.371+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecd0007cc0 con 0x7fece80fdff0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.371+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecd000f040 con 0x7fece80fdff0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.371+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece8115290 con 0x7fece80fdff0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.371+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecd0009640 con 0x7fece80fdff0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.371+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece8115850 con 0x7fece80fdff0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.372+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece8101980 con 0x7fece80fdff0 2024-04-16T11:39:03.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.373+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fecd00097a0 con 0x7fece80fdff0 2024-04-16T11:39:03.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.375+0000 7fecdf400700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7feccc0692a0 0x7feccc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:03.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.375+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fecd0086d30 con 0x7fece80fdff0 2024-04-16T11:39:03.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.375+0000 7fece6a00700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7feccc0692a0 0x7feccc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:03.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.376+0000 7fece6a00700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7feccc0692a0 0x7feccc06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fecd8004f90 tx=0x7fecd8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:03.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.380+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fecd0056d10 con 0x7fece80fdff0 2024-04-16T11:39:03.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:03 smithi045 ceph-mon[36308]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:03.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.603+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fece804c710 con 0x7fece80fdff0 2024-04-16T11:39:03.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.614+0000 7fecdf400700 1 -- 172.21.15.44:0/1317421670 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fecd00568a0 con 0x7fece80fdff0 2024-04-16T11:39:03.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7feccc0692a0 msgr2=0x7feccc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:03.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7feccc0692a0 0x7feccc06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fecd8004f90 tx=0x7fecd8009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:03.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 msgr2=0x7fece81100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:03.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 0x7fece81100c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fecd0002a50 tx=0x7fecd00046b0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 shutdown_connections 2024-04-16T11:39:03.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7feccc0692a0 0x7feccc06b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fece80fce80 0x7fece810fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 --2- 172.21.15.44:0/1317421670 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fece80fdff0 0x7fece81100c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:03.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 >> 172.21.15.44:0/1317421670 conn(0x7fece80f8540 msgr2=0x7fece80f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:03.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 shutdown_connections 2024-04-16T11:39:03.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:03.617+0000 7feced800700 1 -- 172.21.15.44:0/1317421670 wait complete. 2024-04-16T11:39:04.597 INFO:teuthology.run_tasks:Running task print... 2024-04-16T11:39:04.605 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-16T11:39:04.605 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:04.612 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:04.613 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch status' 2024-04-16T11:39:04.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:04 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1317421670' entity='client.admin' 2024-04-16T11:39:04.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:04.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:04.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:39:04.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:04 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:04.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:04 smithi044 ceph-mon[28731]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:04 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1317421670' entity='client.admin' 2024-04-16T11:39:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:39:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:04 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:05.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:04 smithi045 ceph-mon[36308]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:05.104 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:07.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.149+0000 7fb25e200700 1 -- 172.21.15.44:0/432747246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2580fdff0 msgr2=0x7fb2580fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:07.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.149+0000 7fb25e200700 1 --2- 172.21.15.44:0/432747246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2580fdff0 0x7fb2580fe460 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb244004940 tx=0x7fb244009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:07.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.149+0000 7fb25e200700 1 -- 172.21.15.44:0/432747246 shutdown_connections 2024-04-16T11:39:07.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.149+0000 7fb25e200700 1 --2- 172.21.15.44:0/432747246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2580fdff0 0x7fb2580fe460 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.149+0000 7fb25e200700 1 --2- 172.21.15.44:0/432747246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580fce80 0x7fb2580fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.149+0000 7fb25e200700 1 -- 172.21.15.44:0/432747246 >> 172.21.15.44:0/432747246 conn(0x7fb2580f8580 msgr2=0x7fb2580fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.150+0000 7fb25e200700 1 -- 172.21.15.44:0/432747246 shutdown_connections 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.150+0000 7fb25e200700 1 -- 172.21.15.44:0/432747246 wait complete. 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.151+0000 7fb25e200700 1 Processor -- start 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.151+0000 7fb25e200700 1 -- start start 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.151+0000 7fb25e200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580fce80 0x7fb25810daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.151+0000 7fb25e200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 0x7fb258113010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.151+0000 7fb25e200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25810e4a0 con 0x7fb25810dfe0 2024-04-16T11:39:07.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.151+0000 7fb25e200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25810e5e0 con 0x7fb2580fce80 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 0x7fb258113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 0x7fb258113010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49924/0 (socket says 172.21.15.44:49924) 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 -- 172.21.15.44:0/4219401581 learned_addr learned my addr 172.21.15.44:0/4219401581 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb257400700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580fce80 0x7fb25810daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 -- 172.21.15.44:0/4219401581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580fce80 msgr2=0x7fb25810daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580fce80 0x7fb25810daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 -- 172.21.15.44:0/4219401581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244005040 con 0x7fb25810dfe0 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.152+0000 7fb256a00700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 0x7fb258113010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb24400b970 tx=0x7fb244009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.153+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb24400e040 con 0x7fb25810dfe0 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.153+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb244004290 con 0x7fb25810dfe0 2024-04-16T11:39:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.153+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb24401c3f0 con 0x7fb25810dfe0 2024-04-16T11:39:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.153+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb258113550 con 0x7fb25810dfe0 2024-04-16T11:39:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.153+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb258113ac0 con 0x7fb25810dfe0 2024-04-16T11:39:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.154+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25804c710 con 0x7fb25810dfe0 2024-04-16T11:39:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.155+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb24400bbf0 con 0x7fb25810dfe0 2024-04-16T11:39:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.156+0000 7fb24f400700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb23c0692a0 0x7fb23c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.156+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb2440880c0 con 0x7fb25810dfe0 2024-04-16T11:39:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.156+0000 7fb257400700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb23c0692a0 0x7fb23c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.157+0000 7fb257400700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb23c0692a0 0x7fb23c06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb248004990 tx=0x7fb248009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:07.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.161+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb2440581a0 con 0x7fb25810dfe0 2024-04-16T11:39:07.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:06 smithi045 ceph-mon[36308]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.393+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb258102530 con 0x7fb23c0692a0 2024-04-16T11:39:07.392 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:06 smithi044 ceph-mon[28731]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.397+0000 7fb24f400700 1 -- 172.21.15.44:0/4219401581 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb258102530 con 0x7fb23c0692a0 2024-04-16T11:39:07.395 INFO:teuthology.orchestra.run.smithi044.stdout:Backend: cephadm 2024-04-16T11:39:07.395 INFO:teuthology.orchestra.run.smithi044.stdout:Available: Yes 2024-04-16T11:39:07.395 INFO:teuthology.orchestra.run.smithi044.stdout:Paused: No 2024-04-16T11:39:07.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb23c0692a0 msgr2=0x7fb23c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:07.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb23c0692a0 0x7fb23c06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb248004990 tx=0x7fb248009ee0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 msgr2=0x7fb258113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:07.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 0x7fb258113010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb24400b970 tx=0x7fb244009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:07.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 shutdown_connections 2024-04-16T11:39:07.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb23c0692a0 0x7fb23c06b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2580fce80 0x7fb25810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 --2- 172.21.15.44:0/4219401581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25810dfe0 0x7fb258113010 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:07.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 >> 172.21.15.44:0/4219401581 conn(0x7fb2580f8580 msgr2=0x7fb2580f9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:07.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 shutdown_connections 2024-04-16T11:39:07.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:07.400+0000 7fb25e200700 1 -- 172.21.15.44:0/4219401581 wait complete. 2024-04-16T11:39:08.109 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-16T11:39:08.599 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:09.067 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:08 smithi044 ceph-mon[28731]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:09.067 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:08 smithi044 ceph-mon[28731]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:08 smithi045 ceph-mon[36308]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:08 smithi045 ceph-mon[36308]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 -- 172.21.15.44:0/1451277622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff334100070 msgr2=0x7ff3341004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 --2- 172.21.15.44:0/1451277622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff334100070 0x7ff3341004e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff320004940 tx=0x7ff320009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 -- 172.21.15.44:0/1451277622 shutdown_connections 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 --2- 172.21.15.44:0/1451277622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff334100070 0x7ff3341004e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 --2- 172.21.15.44:0/1451277622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3340fef00 0x7ff3340ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 -- 172.21.15.44:0/1451277622 >> 172.21.15.44:0/1451277622 conn(0x7ff3340fa790 msgr2=0x7ff3340fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 -- 172.21.15.44:0/1451277622 shutdown_connections 2024-04-16T11:39:10.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.681+0000 7ff339400700 1 -- 172.21.15.44:0/1451277622 wait complete. 2024-04-16T11:39:10.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.682+0000 7ff339400700 1 Processor -- start 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.682+0000 7ff339400700 1 -- start start 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff339400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 0x7ff33410fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff339400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff334100070 0x7ff33410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff339400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3341105a0 con 0x7ff3340fef00 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff339400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334110df0 con 0x7ff334100070 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff333e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 0x7ff33410fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff333e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 0x7ff33410fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49934/0 (socket says 172.21.15.44:49934) 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff333e00700 1 -- 172.21.15.44:0/1309317388 learned_addr learned my addr 172.21.15.44:0/1309317388 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.683+0000 7ff333400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff334100070 0x7ff33410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff333e00700 1 -- 172.21.15.44:0/1309317388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff334100070 msgr2=0x7ff33410ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff333e00700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff334100070 0x7ff33410ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:10.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff333e00700 1 -- 172.21.15.44:0/1309317388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320005040 con 0x7ff3340fef00 2024-04-16T11:39:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff333e00700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 0x7ff33410fa60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff32400c830 tx=0x7ff324004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32400a760 con 0x7ff3340fef00 2024-04-16T11:39:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff32400ad90 con 0x7ff3340fef00 2024-04-16T11:39:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3240113a0 con 0x7ff3340fef00 2024-04-16T11:39:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.684+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3341110d0 con 0x7ff3340fef00 2024-04-16T11:39:10.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.685+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334111610 con 0x7ff3340fef00 2024-04-16T11:39:10.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.685+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33405f5f0 con 0x7ff3340fef00 2024-04-16T11:39:10.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.687+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff324012400 con 0x7ff3340fef00 2024-04-16T11:39:10.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.688+0000 7ff32be00700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff3180692b0 0x7ff31806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:10.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.688+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff324086eb0 con 0x7ff3340fef00 2024-04-16T11:39:10.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.688+0000 7ff333400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff3180692b0 0x7ff31806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:10.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.689+0000 7ff333400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff3180692b0 0x7ff31806b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff320004f90 tx=0x7ff320009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:10.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.693+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff324056f90 con 0x7ff3340fef00 2024-04-16T11:39:10.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.919+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff334111a70 con 0x7ff3180692b0 2024-04-16T11:39:10.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.937+0000 7ff32be00700 1 -- 172.21.15.44:0/1309317388 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7ff334111a70 con 0x7ff3180692b0 2024-04-16T11:39:10.935 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:39:10.935 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (4m) 2m ago 6m 14.2M - 0.25.0 c8568f914cd2 8136acabd663 2024-04-16T11:39:10.935 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (7m) 2m ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:39:10.935 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (5m) 82s ago 5m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:39:10.935 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (4m) 2m ago 6m 73.8M - 9.4.7 954c08fa6188 afde3dcbfd10 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:9283 running (8m) 2m ago 8m 473M - 17.2.7-847-g45317f95 b333581ccdf5 53edec877549 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (5m) 82s ago 5m 431M - 17.2.7-847-g45317f95 b333581ccdf5 f6be9ec13da1 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (8m) 2m ago 8m 38.9M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (5m) 82s ago 5m 37.6M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (6m) 2m ago 6m 7495k - 1.5.0 0da6a335fe13 dbe9480d7dd0 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (5m) 82s ago 5m 7772k - 1.5.0 0da6a335fe13 e37d89867e75 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (4m) 2m ago 4m 34.9M 4290M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (3m) 2m ago 3m 36.7M 4290M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (3m) 2m ago 3m 10.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (2m) 82s ago 2m 38.3M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (2m) 82s ago 2m 35.7M 5314M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (88s) 82s ago 87s 10.5M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:39:10.936 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (4m) 2m ago 6m 34.7M - 2.43.0 a07b618ecd1d 99d239fdb386 2024-04-16T11:39:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.939+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff3180692b0 msgr2=0x7ff31806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.939+0000 7ff339400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff3180692b0 0x7ff31806b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff320004f90 tx=0x7ff320009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 msgr2=0x7ff33410fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 0x7ff33410fa60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff32400c830 tx=0x7ff324004470 comp rx=0 tx=0).stop 2024-04-16T11:39:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 shutdown_connections 2024-04-16T11:39:10.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff3180692b0 0x7ff31806b760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3340fef00 0x7ff33410fa60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 --2- 172.21.15.44:0/1309317388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff334100070 0x7ff33410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 >> 172.21.15.44:0/1309317388 conn(0x7ff3340fa790 msgr2=0x7ff3340fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 shutdown_connections 2024-04-16T11:39:10.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:10.940+0000 7ff339400700 1 -- 172.21.15.44:0/1309317388 wait complete. 2024-04-16T11:39:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:11 smithi044 ceph-mon[28731]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:11 smithi045 ceph-mon[36308]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:11.842 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-16T11:39:12.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:12 smithi044 ceph-mon[28731]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:12.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:12 smithi045 ceph-mon[36308]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:12.357 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:13.076 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:13 smithi044 ceph-mon[28731]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:13.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:13 smithi045 ceph-mon[36308]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:14.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 -- 172.21.15.44:0/1976436630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 msgr2=0x7f69b8102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:14.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1976436630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8102160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f69a4004940 tx=0x7f69a4009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:14.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 -- 172.21.15.44:0/1976436630 shutdown_connections 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1976436630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8102160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1976436630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b80662f0 0x7f69b80ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 -- 172.21.15.44:0/1976436630 >> 172.21.15.44:0/1976436630 conn(0x7f69b80f7270 msgr2=0x7f69b80f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 -- 172.21.15.44:0/1976436630 shutdown_connections 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.461+0000 7f69b7e00700 1 -- 172.21.15.44:0/1976436630 wait complete. 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.462+0000 7f69b7e00700 1 Processor -- start 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.462+0000 7f69b7e00700 1 -- start start 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b7e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b80662f0 0x7f69b810fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b7e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8110600 con 0x7f69b80ffd40 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b7e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8110e50 con 0x7f69b80662f0 2024-04-16T11:39:14.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51590/0 (socket says 172.21.15.44:51590) 2024-04-16T11:39:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b6000700 1 -- 172.21.15.44:0/1475355513 learned_addr learned my addr 172.21.15.44:0/1475355513 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b6a00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b80662f0 0x7f69b810fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b6000700 1 -- 172.21.15.44:0/1475355513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b80662f0 msgr2=0x7f69b810fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.463+0000 7f69b6000700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b80662f0 0x7f69b810fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69b6000700 1 -- 172.21.15.44:0/1475355513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a4005040 con 0x7f69b80ffd40 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69b6000700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8110000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f69a400b3b0 tx=0x7f69a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a400e040 con 0x7f69b80ffd40 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69a4004290 con 0x7f69b80ffd40 2024-04-16T11:39:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b81110d0 con 0x7f69b80ffd40 2024-04-16T11:39:14.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a401c630 con 0x7f69b80ffd40 2024-04-16T11:39:14.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.464+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b8111610 con 0x7f69b80ffd40 2024-04-16T11:39:14.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.466+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b805f5f0 con 0x7f69b80ffd40 2024-04-16T11:39:14.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.467+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f69a400bbf0 con 0x7f69b80ffd40 2024-04-16T11:39:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.467+0000 7f69aea00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f699c0692b0 0x7f699c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.468+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f69a4088350 con 0x7f69b80ffd40 2024-04-16T11:39:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.468+0000 7f69b6a00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f699c0692b0 0x7f699c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:14.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.468+0000 7f69b6a00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f699c0692b0 0x7f699c06b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f69a8004550 tx=0x7f69a8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:14.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.472+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f69a40583b0 con 0x7f69b80ffd40 2024-04-16T11:39:14.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.701+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f69b81118f0 con 0x7f699c0692b0 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.715+0000 7f69aea00700 1 -- 172.21.15.44:0/1475355513 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7f69b81118f0 con 0x7f699c0692b0 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager ?:9093,9094 1/1 3m ago 7m count:1 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stdout:crash 2/2 3m ago 7m * 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stdout:grafana ?:3000 1/1 3m ago 7m count:1 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stdout:mgr 2/2 3m ago 7m count:2 2024-04-16T11:39:14.713 INFO:teuthology.orchestra.run.smithi044.stdout:mon 2/2 3m ago 6m smithi044:172.21.15.44=smithi044;smithi045:172.21.15.45=smithi045;count:2 2024-04-16T11:39:14.714 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter ?:9100 2/2 3m ago 7m * 2024-04-16T11:39:14.714 INFO:teuthology.orchestra.run.smithi044.stdout:osd 6 3m ago - 2024-04-16T11:39:14.714 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus ?:9095 1/1 3m ago 7m count:1 2024-04-16T11:39:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f699c0692b0 msgr2=0x7f699c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f699c0692b0 0x7f699c06b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f69a8004550 tx=0x7f69a8009040 comp rx=0 tx=0).stop 2024-04-16T11:39:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 msgr2=0x7f69b8110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8110000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f69a400b3b0 tx=0x7f69a4009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 shutdown_connections 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f699c0692b0 0x7f699c06b760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69b80662f0 0x7f69b810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 --2- 172.21.15.44:0/1475355513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69b80ffd40 0x7f69b8110000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 >> 172.21.15.44:0/1475355513 conn(0x7f69b80f7270 msgr2=0x7f69b80fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 shutdown_connections 2024-04-16T11:39:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:14.718+0000 7f69b7e00700 1 -- 172.21.15.44:0/1475355513 wait complete. 2024-04-16T11:39:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:15 smithi045 ceph-mon[36308]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:15.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:15 smithi044 ceph-mon[28731]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:15.543 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-16T11:39:16.046 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:16.327 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:16 smithi044 ceph-mon[28731]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:16.327 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:16 smithi045 ceph-mon[36308]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:17.216 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:17 smithi044 ceph-mon[28731]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:17.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:17 smithi045 ceph-mon[36308]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.207+0000 7f4ed0200700 1 -- 172.21.15.44:0/2287535900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec80fff30 msgr2=0x7f4ec8102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.207+0000 7f4ed0200700 1 --2- 172.21.15.44:0/2287535900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec80fff30 0x7f4ec8102350 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4004940 tx=0x7f4eb4009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.208+0000 7f4ed0200700 1 -- 172.21.15.44:0/2287535900 shutdown_connections 2024-04-16T11:39:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.208+0000 7f4ed0200700 1 --2- 172.21.15.44:0/2287535900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec80fff30 0x7f4ec8102350 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.208+0000 7f4ed0200700 1 --2- 172.21.15.44:0/2287535900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8066f80 0x7f4ec80ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:18.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.208+0000 7f4ed0200700 1 -- 172.21.15.44:0/2287535900 >> 172.21.15.44:0/2287535900 conn(0x7f4ec80f7460 msgr2=0x7f4ec80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.208+0000 7f4ed0200700 1 -- 172.21.15.44:0/2287535900 shutdown_connections 2024-04-16T11:39:18.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.208+0000 7f4ed0200700 1 -- 172.21.15.44:0/2287535900 wait complete. 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ed0200700 1 Processor -- start 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ed0200700 1 -- start start 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ed0200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 0x7f4ec80fd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ed0200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec80fff30 0x7f4ec80fb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ed0200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80fd840 con 0x7f4ec8066f80 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ed0200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80fbe30 con 0x7f4ec80fff30 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.209+0000 7f4ecd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 0x7f4ec80fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:18.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 0x7f4ec80fd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51596/0 (socket says 172.21.15.44:51596) 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecd800700 1 -- 172.21.15.44:0/421233406 learned_addr learned my addr 172.21.15.44:0/421233406 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecce00700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec80fff30 0x7f4ec80fb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecd800700 1 -- 172.21.15.44:0/421233406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec80fff30 msgr2=0x7f4ec80fb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecd800700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec80fff30 0x7f4ec80fb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecd800700 1 -- 172.21.15.44:0/421233406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb4005040 con 0x7f4ec8066f80 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecce00700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec80fff30 0x7f4ec80fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:39:18.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.210+0000 7f4ecd800700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 0x7f4ec80fd240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc002a50 tx=0x7f4ebc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:18.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.211+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ebc007cc0 con 0x7f4ec8066f80 2024-04-16T11:39:18.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.211+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ebc00f040 con 0x7f4ec8066f80 2024-04-16T11:39:18.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.211+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ebc00da40 con 0x7f4ec8066f80 2024-04-16T11:39:18.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.211+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec80fc0e0 con 0x7f4ec8066f80 2024-04-16T11:39:18.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.211+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec80fc6a0 con 0x7f4ec8066f80 2024-04-16T11:39:18.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.212+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec804c710 con 0x7f4ec8066f80 2024-04-16T11:39:18.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.214+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f4ebc0097c0 con 0x7f4ec8066f80 2024-04-16T11:39:18.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.215+0000 7f4ec5600700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4eb00692a0 0x7f4eb006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:18.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.215+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ebc086b20 con 0x7f4ec8066f80 2024-04-16T11:39:18.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.215+0000 7f4ecce00700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4eb00692a0 0x7f4eb006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:18.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.216+0000 7f4ecce00700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4eb00692a0 0x7f4eb006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4000c00 tx=0x7f4eb4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:18.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.220+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4ebc056b00 con 0x7f4ec8066f80 2024-04-16T11:39:18.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.445+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ec80f8fa0 con 0x7f4eb00692a0 2024-04-16T11:39:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.447+0000 7f4ec5600700 1 -- 172.21.15.44:0/421233406 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f4ec80f8fa0 con 0x7f4eb00692a0 2024-04-16T11:39:18.475 INFO:teuthology.orchestra.run.smithi044.stdout:HOST ADDR LABELS STATUS 2024-04-16T11:39:18.475 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 172.21.15.44 2024-04-16T11:39:18.475 INFO:teuthology.orchestra.run.smithi044.stdout:smithi045 172.21.15.45 2024-04-16T11:39:18.475 INFO:teuthology.orchestra.run.smithi044.stdout:2 hosts in cluster 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4eb00692a0 msgr2=0x7f4eb006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4eb00692a0 0x7f4eb006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4000c00 tx=0x7f4eb4009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 msgr2=0x7f4ec80fd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 0x7f4ec80fd240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc002a50 tx=0x7f4ebc004600 comp rx=0 tx=0).stop 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 shutdown_connections 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f4eb00692a0 0x7f4eb006b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec8066f80 0x7f4ec80fd240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 --2- 172.21.15.44:0/421233406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec80fff30 0x7f4ec80fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 >> 172.21.15.44:0/421233406 conn(0x7f4ec80f7460 msgr2=0x7f4ec80f7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 shutdown_connections 2024-04-16T11:39:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:18.450+0000 7f4ed0200700 1 -- 172.21.15.44:0/421233406 wait complete. 2024-04-16T11:39:19.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:19 smithi044 ceph-mon[28731]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:19 smithi045 ceph-mon[36308]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:19.340 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-16T11:39:19.837 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:20.311 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:20 smithi044 ceph-mon[28731]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:20 smithi045 ceph-mon[36308]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:21.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:21 smithi045 ceph-mon[36308]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:21.409 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:21 smithi044 ceph-mon[28731]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 -- 172.21.15.44:0/289197405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2941002b0 msgr2=0x7fa294100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 --2- 172.21.15.44:0/289197405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2941002b0 0x7fa294100720 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa280004940 tx=0x7fa280009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 -- 172.21.15.44:0/289197405 shutdown_connections 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 --2- 172.21.15.44:0/289197405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2941002b0 0x7fa294100720 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 --2- 172.21.15.44:0/289197405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2940ff000 0x7fa2940ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 -- 172.21.15.44:0/289197405 >> 172.21.15.44:0/289197405 conn(0x7fa2940fa790 msgr2=0x7fa2940fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 -- 172.21.15.44:0/289197405 shutdown_connections 2024-04-16T11:39:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.884+0000 7fa29a400700 1 -- 172.21.15.44:0/289197405 wait complete. 2024-04-16T11:39:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.885+0000 7fa29a400700 1 Processor -- start 2024-04-16T11:39:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.885+0000 7fa29a400700 1 -- start start 2024-04-16T11:39:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa29a400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 0x7fa29410fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa29a400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2941002b0 0x7fa294110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa29a400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294110670 con 0x7fa2940ff000 2024-04-16T11:39:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa29a400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2941107e0 con 0x7fa2941002b0 2024-04-16T11:39:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa293400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 0x7fa29410fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa293400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 0x7fa29410fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60094/0 (socket says 172.21.15.44:60094) 2024-04-16T11:39:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa293400700 1 -- 172.21.15.44:0/3875172909 learned_addr learned my addr 172.21.15.44:0/3875172909 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.886+0000 7fa292a00700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2941002b0 0x7fa294110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa293400700 1 -- 172.21.15.44:0/3875172909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2941002b0 msgr2=0x7fa294110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa293400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2941002b0 0x7fa294110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa293400700 1 -- 172.21.15.44:0/3875172909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280005040 con 0x7fa2940ff000 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa292a00700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2941002b0 0x7fa294110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa293400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 0x7fa29410fb30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa28400c830 tx=0x7fa284004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28400a760 con 0x7fa2940ff000 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa28400ad90 con 0x7fa2940ff000 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.887+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa294115240 con 0x7fa2940ff000 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.888+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2840113a0 con 0x7fa2940ff000 2024-04-16T11:39:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.888+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa294115800 con 0x7fa2940ff000 2024-04-16T11:39:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.889+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29404c710 con 0x7fa2940ff000 2024-04-16T11:39:21.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.890+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa284012400 con 0x7fa2940ff000 2024-04-16T11:39:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.891+0000 7fa28b400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2780692a0 0x7fa27806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:21.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.891+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa284086ee0 con 0x7fa2940ff000 2024-04-16T11:39:21.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.892+0000 7fa292a00700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2780692a0 0x7fa27806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:21.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.892+0000 7fa292a00700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2780692a0 0x7fa27806b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa28000b020 tx=0x7fa280009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:21.896+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa284056fc0 con 0x7fa2940ff000 2024-04-16T11:39:22.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.123+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2941045c0 con 0x7fa2780692a0 2024-04-16T11:39:22.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.127+0000 7fa28b400700 1 -- 172.21.15.44:0/3875172909 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fa2941045c0 con 0x7fa2780692a0 2024-04-16T11:39:22.125 INFO:teuthology.orchestra.run.smithi044.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-16T11:39:22.125 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001K400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-16T11:39:22.125 INFO:teuthology.orchestra.run.smithi044.stdout:smithi045 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75120098480DGN 447G No 88s ago Has a FileSystem, LVM detected 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.129+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2780692a0 msgr2=0x7fa27806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.129+0000 7fa29a400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2780692a0 0x7fa27806b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa28000b020 tx=0x7fa280009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 msgr2=0x7fa29410fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 0x7fa29410fb30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa28400c830 tx=0x7fa284004470 comp rx=0 tx=0).stop 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 shutdown_connections 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fa2780692a0 0x7fa27806b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2940ff000 0x7fa29410fb30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 --2- 172.21.15.44:0/3875172909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2941002b0 0x7fa294110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 >> 172.21.15.44:0/3875172909 conn(0x7fa2940fa790 msgr2=0x7fa2940fbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:22.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 shutdown_connections 2024-04-16T11:39:22.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:22.130+0000 7fa29a400700 1 -- 172.21.15.44:0/3875172909 wait complete. 2024-04-16T11:39:22.943 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:22.953 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:22.953 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-16T11:39:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:23 smithi044 ceph-mon[28731]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:23.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:23 smithi044 ceph-mon[28731]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:23.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:23 smithi045 ceph-mon[36308]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:23.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:23 smithi045 ceph-mon[36308]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:23.459 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:25.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:25 smithi045 ceph-mon[36308]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:25.415 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:25 smithi044 ceph-mon[28731]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 -- 172.21.15.44:0/3011519187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef4100070 msgr2=0x7fbef41004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 --2- 172.21.15.44:0/3011519187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef4100070 0x7fbef41004e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbee0004940 tx=0x7fbee0009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 -- 172.21.15.44:0/3011519187 shutdown_connections 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 --2- 172.21.15.44:0/3011519187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef4100070 0x7fbef41004e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 --2- 172.21.15.44:0/3011519187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbef40fef00 0x7fbef40ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 -- 172.21.15.44:0/3011519187 >> 172.21.15.44:0/3011519187 conn(0x7fbef40fa790 msgr2=0x7fbef40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 -- 172.21.15.44:0/3011519187 shutdown_connections 2024-04-16T11:39:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.593+0000 7fbefa000700 1 -- 172.21.15.44:0/3011519187 wait complete. 2024-04-16T11:39:25.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.594+0000 7fbefa000700 1 Processor -- start 2024-04-16T11:39:25.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.594+0000 7fbefa000700 1 -- start start 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbefa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 0x7fbef410fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbefa000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbef4100070 0x7fbef410ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbefa000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef41105b0 con 0x7fbef40fef00 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbefa000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4110e00 con 0x7fbef4100070 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbef8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 0x7fbef410fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbef8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 0x7fbef410fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60116/0 (socket says 172.21.15.44:60116) 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbef8c00700 1 -- 172.21.15.44:0/120318327 learned_addr learned my addr 172.21.15.44:0/120318327 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.595+0000 7fbef3e00700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbef4100070 0x7fbef410ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef8c00700 1 -- 172.21.15.44:0/120318327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbef4100070 msgr2=0x7fbef410ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef8c00700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbef4100070 0x7fbef410ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:25.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef8c00700 1 -- 172.21.15.44:0/120318327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee0005040 con 0x7fbef40fef00 2024-04-16T11:39:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef8c00700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 0x7fbef410fa70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c8c0 tx=0x7fbee80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee800a760 con 0x7fbef40fef00 2024-04-16T11:39:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbee800ad90 con 0x7fbef40fef00 2024-04-16T11:39:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef41110e0 con 0x7fbef40fef00 2024-04-16T11:39:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee80113a0 con 0x7fbef40fef00 2024-04-16T11:39:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.596+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef4111620 con 0x7fbef40fef00 2024-04-16T11:39:25.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.598+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef40fe290 con 0x7fbef40fef00 2024-04-16T11:39:25.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.599+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fbee8012400 con 0x7fbef40fef00 2024-04-16T11:39:25.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.600+0000 7fbef0c00700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbedc0692a0 0x7fbedc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:25.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.600+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbee8086ed0 con 0x7fbef40fef00 2024-04-16T11:39:25.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.601+0000 7fbef3e00700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbedc0692a0 0x7fbedc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:25.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.601+0000 7fbef3e00700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbedc0692a0 0x7fbedc06b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009fd0 tx=0x7fbee0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:25.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.605+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbee8056fb0 con 0x7fbef40fef00 2024-04-16T11:39:25.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:25.853+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbef4111ac0 con 0x7fbedc0692a0 2024-04-16T11:39:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:26 smithi045 ceph-mon[36308]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:26 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T11:39:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:26 smithi045 ceph-mon[36308]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:26.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:26 smithi044 ceph-mon[28731]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:39:26.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:26 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T11:39:26.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:26 smithi044 ceph-mon[28731]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:27.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.536+0000 7fbef0c00700 1 -- 172.21.15.44:0/120318327 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbef4111ac0 con 0x7fbedc0692a0 2024-04-16T11:39:27.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbedc0692a0 msgr2=0x7fbedc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:27.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbedc0692a0 0x7fbedc06b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009fd0 tx=0x7fbee0009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:27.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 msgr2=0x7fbef410fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 0x7fbef410fa70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c8c0 tx=0x7fbee80044c0 comp rx=0 tx=0).stop 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 shutdown_connections 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fbedc0692a0 0x7fbedc06b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbef40fef00 0x7fbef410fa70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 --2- 172.21.15.44:0/120318327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbef4100070 0x7fbef410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 >> 172.21.15.44:0/120318327 conn(0x7fbef40fa790 msgr2=0x7fbef40fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 shutdown_connections 2024-04-16T11:39:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:27.539+0000 7fbefa000700 1 -- 172.21.15.44:0/120318327 wait complete. 2024-04-16T11:39:27.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:27 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T11:39:27.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:27 smithi045 ceph-mon[36308]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T11:39:27.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:27 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T11:39:27.952 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:27 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T11:39:27.952 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:27 smithi044 ceph-mon[28731]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T11:39:27.952 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:27 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T11:39:27.952 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:27 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[28727]: 2024-04-16T11:39:27.510+0000 7f1121a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:39:28.385 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T11:39:28.577 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: fsmap cephfs:0 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: Saving service mds.cephfs spec with placement count:4 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:28.578 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:39:28.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:28.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:28.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:28.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:28.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: Deploying daemon mds.cephfs.smithi044.ufhhlc on smithi044 2024-04-16T11:39:28.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:28 smithi044 ceph-mon[28731]: pgmap v217: 65 pgs: 5 creating+peering, 1 active+clean, 59 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:28.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: fsmap cephfs:0 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: Saving service mds.cephfs spec with placement count:4 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:28.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:28.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: Deploying daemon mds.cephfs.smithi044.ufhhlc on smithi044 2024-04-16T11:39:28.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:28 smithi045 ceph-mon[36308]: pgmap v217: 65 pgs: 5 creating+peering, 1 active+clean, 59 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:29.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:29 smithi044 ceph-mon[28731]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T11:39:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:29.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:29 smithi045 ceph-mon[36308]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T11:39:30.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:30 smithi044 ceph-mon[28731]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T11:39:30.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:30 smithi044 ceph-mon[28731]: pgmap v220: 65 pgs: 5 creating+peering, 28 active+clean, 32 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:30.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:30 smithi045 ceph-mon[36308]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T11:39:30.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:30 smithi045 ceph-mon[36308]: pgmap v220: 65 pgs: 5 creating+peering, 28 active+clean, 32 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:31.953 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:31.953 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:32.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:31 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:31 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 -- 172.21.15.44:0/882928540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 msgr2=0x7f71a40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 --2- 172.21.15.44:0/882928540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a40fe550 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f718c004940 tx=0x7f718c009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 -- 172.21.15.44:0/882928540 shutdown_connections 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 --2- 172.21.15.44:0/882928540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a40fe550 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 --2- 172.21.15.44:0/882928540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71a40fcef0 0x7f71a40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 -- 172.21.15.44:0/882928540 >> 172.21.15.44:0/882928540 conn(0x7f71a40f8530 msgr2=0x7f71a40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 -- 172.21.15.44:0/882928540 shutdown_connections 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.567+0000 7f71a8a00700 1 -- 172.21.15.44:0/882928540 wait complete. 2024-04-16T11:39:32.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.568+0000 7f71a8a00700 1 Processor -- start 2024-04-16T11:39:32.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.568+0000 7f71a8a00700 1 -- start start 2024-04-16T11:39:32.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.568+0000 7f71a8a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71a40fcef0 0x7f71a410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:32.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.568+0000 7f71a8a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a41101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.568+0000 7f71a8a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a41107a0 con 0x7f71a40fe0e0 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.568+0000 7f71a8a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a4110ff0 con 0x7f71a40fcef0 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f7199400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a41101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f7199400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a41101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39570/0 (socket says 172.21.15.44:39570) 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f7199400700 1 -- 172.21.15.44:0/2403079515 learned_addr learned my addr 172.21.15.44:0/2403079515 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f71a1e00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71a40fcef0 0x7f71a410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f7199400700 1 -- 172.21.15.44:0/2403079515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71a40fcef0 msgr2=0x7f71a410fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f7199400700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71a40fcef0 0x7f71a410fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.569+0000 7f7199400700 1 -- 172.21.15.44:0/2403079515 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c005040 con 0x7f71a40fe0e0 2024-04-16T11:39:32.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.570+0000 7f7199400700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a41101a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f718c009150 tx=0x7f718c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.570+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f718c00e040 con 0x7f71a40fe0e0 2024-04-16T11:39:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.570+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f718c00bcc0 con 0x7f71a40fe0e0 2024-04-16T11:39:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.570+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f718c01c630 con 0x7f71a40fe0e0 2024-04-16T11:39:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.570+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a4111270 con 0x7f71a40fe0e0 2024-04-16T11:39:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.570+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a41117b0 con 0x7f71a40fe0e0 2024-04-16T11:39:32.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.571+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a4101a70 con 0x7f71a40fe0e0 2024-04-16T11:39:32.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.572+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f718c013420 con 0x7f71a40fe0e0 2024-04-16T11:39:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.573+0000 7f719aa00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f71880692a0 0x7f718806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.573+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f718c087f40 con 0x7f71a40fe0e0 2024-04-16T11:39:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.573+0000 7f71a1e00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f71880692a0 0x7f718806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.574+0000 7f71a1e00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f71880692a0 0x7f718806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7194009fd0 tx=0x7f7194009f00 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:32.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.579+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f718c057ce0 con 0x7f71a40fe0e0 2024-04-16T11:39:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.842+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f71a405f5f0 con 0x7f71a40fe0e0 2024-04-16T11:39:32.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.843+0000 7f719aa00700 1 -- 172.21.15.44:0/2403079515 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7f718c057870 con 0x7f71a40fe0e0 2024-04-16T11:39:32.842 INFO:teuthology.orchestra.run.smithi044.stdout:e4 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 4 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:31.930026+0000 2024-04-16T11:39:32.843 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:39:32.844 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 0 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:creating seq 1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:32.845 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.847+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f71880692a0 msgr2=0x7f718806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.847+0000 7f71a8a00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f71880692a0 0x7f718806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7194009fd0 tx=0x7f7194009f00 comp rx=0 tx=0).stop 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.847+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 msgr2=0x7f71a41101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a41101a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f718c009150 tx=0x7f718c009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 shutdown_connections 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f71880692a0 0x7f718806b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:32.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71a40fcef0 0x7f71a410fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 --2- 172.21.15.44:0/2403079515 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71a40fe0e0 0x7f71a41101a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 >> 172.21.15.44:0/2403079515 conn(0x7f71a40f8530 msgr2=0x7f71a40f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 shutdown_connections 2024-04-16T11:39:32.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:32.848+0000 7f71a8a00700 1 -- 172.21.15.44:0/2403079515 wait complete. 2024-04-16T11:39:32.848 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 4 2024-04-16T11:39:33.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] up:boot 2024-04-16T11:39:33.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: daemon mds.cephfs.smithi044.ufhhlc assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T11:39:33.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:39:33.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: Cluster is now healthy 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: fsmap cephfs:0 1 up:standby 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: pgmap v221: 65 pgs: 5 creating+peering, 56 active+clean, 4 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:creating} 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: daemon mds.cephfs.smithi044.ufhhlc is now active in filesystem cephfs as rank 0 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: Deploying daemon mds.cephfs.smithi045.noofsd on smithi045 2024-04-16T11:39:33.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:32 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2403079515' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] up:boot 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: daemon mds.cephfs.smithi044.ufhhlc assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: Cluster is now healthy 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: fsmap cephfs:0 1 up:standby 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: pgmap v221: 65 pgs: 5 creating+peering, 56 active+clean, 4 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:creating} 2024-04-16T11:39:33.255 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: daemon mds.cephfs.smithi044.ufhhlc is now active in filesystem cephfs as rank 0 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: Deploying daemon mds.cephfs.smithi045.noofsd on smithi045 2024-04-16T11:39:33.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:32 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2403079515' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:39:33.717 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:33.727 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:33.727 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-04-16T11:39:34.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:33 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] up:active 2024-04-16T11:39:34.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:33 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2024-04-16T11:39:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:33 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] up:active 2024-04-16T11:39:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:33 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2024-04-16T11:39:34.242 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:34.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:34 smithi044 ceph-mon[28731]: pgmap v222: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 159 B/s wr, 2 op/s 2024-04-16T11:39:35.230 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:34 smithi045 ceph-mon[36308]: pgmap v222: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 159 B/s wr, 2 op/s 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 -- 172.21.15.44:0/2966808817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c1002b0 msgr2=0x7fb25c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 --2- 172.21.15.44:0/2966808817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c1002b0 0x7fb25c100720 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb248004940 tx=0x7fb248009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 -- 172.21.15.44:0/2966808817 shutdown_connections 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 --2- 172.21.15.44:0/2966808817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c1002b0 0x7fb25c100720 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 --2- 172.21.15.44:0/2966808817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c0ff000 0x7fb25c0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 -- 172.21.15.44:0/2966808817 >> 172.21.15.44:0/2966808817 conn(0x7fb25c0fa950 msgr2=0x7fb25c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 -- 172.21.15.44:0/2966808817 shutdown_connections 2024-04-16T11:39:36.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.544+0000 7fb263a00700 1 -- 172.21.15.44:0/2966808817 wait complete. 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.545+0000 7fb263a00700 1 Processor -- start 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.545+0000 7fb263a00700 1 -- start start 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb263a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 0x7fb25c10fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb263a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c1002b0 0x7fb25c110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb263a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c110770 con 0x7fb25c0ff000 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb263a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c110fc0 con 0x7fb25c1002b0 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb261000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 0x7fb25c10fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:36.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb261000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 0x7fb25c10fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39588/0 (socket says 172.21.15.44:39588) 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.546+0000 7fb261000700 1 -- 172.21.15.44:0/3230453540 learned_addr learned my addr 172.21.15.44:0/3230453540 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb25be00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c1002b0 0x7fb25c110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb261000700 1 -- 172.21.15.44:0/3230453540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c1002b0 msgr2=0x7fb25c110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb261000700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c1002b0 0x7fb25c110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb261000700 1 -- 172.21.15.44:0/3230453540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248005040 con 0x7fb25c0ff000 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb25be00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c1002b0 0x7fb25c110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:39:36.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb261000700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 0x7fb25c10fc30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb25000c830 tx=0x7fb250004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:36.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb25000a760 con 0x7fb25c0ff000 2024-04-16T11:39:36.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c1112a0 con 0x7fb25c0ff000 2024-04-16T11:39:36.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb25000ad90 con 0x7fb25c0ff000 2024-04-16T11:39:36.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.547+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2500113a0 con 0x7fb25c0ff000 2024-04-16T11:39:36.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.548+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c1117e0 con 0x7fb25c0ff000 2024-04-16T11:39:36.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.549+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c05f5f0 con 0x7fb25c0ff000 2024-04-16T11:39:36.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.551+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb250012400 con 0x7fb25c0ff000 2024-04-16T11:39:36.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.551+0000 7fb258c00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb2440692b0 0x7fb24406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:36.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.552+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb250087210 con 0x7fb25c0ff000 2024-04-16T11:39:36.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.552+0000 7fb25be00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb2440692b0 0x7fb24406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:36.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.552+0000 7fb25be00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb2440692b0 0x7fb24406b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb24800b7a0 tx=0x7fb248009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.557+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb250056fb0 con 0x7fb25c0ff000 2024-04-16T11:39:36.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:36.891+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fb25c111c00 con 0x7fb25c0ff000 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: Deploying daemon mds.cephfs.smithi044.hlafhv on smithi044 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: pgmap v223: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:boot 2024-04-16T11:39:37.023 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 1 up:standby 2024-04-16T11:39:37.024 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:39:37.024 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:36 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 1 up:standby 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: Deploying daemon mds.cephfs.smithi044.hlafhv on smithi044 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: pgmap v223: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:boot 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 1 up:standby 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:39:37.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:36 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 1 up:standby 2024-04-16T11:39:37.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.832+0000 7fb258c00700 1 -- 172.21.15.44:0/3230453540 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb25005c070 con 0x7fb25c0ff000 2024-04-16T11:39:37.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.836+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb2440692b0 msgr2=0x7fb24406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:37.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.836+0000 7fb263a00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb2440692b0 0x7fb24406b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb24800b7a0 tx=0x7fb248009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:37.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.836+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 msgr2=0x7fb25c10fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:37.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.836+0000 7fb263a00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 0x7fb25c10fc30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb25000c830 tx=0x7fb250004470 comp rx=0 tx=0).stop 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 shutdown_connections 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb2440692b0 0x7fb24406b760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb25c0ff000 0x7fb25c10fc30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 --2- 172.21.15.44:0/3230453540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb25c1002b0 0x7fb25c110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 >> 172.21.15.44:0/3230453540 conn(0x7fb25c0fa950 msgr2=0x7fb25c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 shutdown_connections 2024-04-16T11:39:37.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:37.837+0000 7fb263a00700 1 -- 172.21.15.44:0/3230453540 wait complete. 2024-04-16T11:39:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:37 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3230453540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-16T11:39:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:37 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3230453540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-16T11:39:39.077 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:38 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3230453540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-16T11:39:39.077 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:38 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 1 up:standby 2024-04-16T11:39:39.077 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:38 smithi044 ceph-mon[28731]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-16T11:39:39.198 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:39.209 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:39.209 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-16T11:39:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:38 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3230453540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-16T11:39:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:38 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 1 up:standby 2024-04-16T11:39:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:38 smithi045 ceph-mon[36308]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-16T11:39:39.956 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:boot 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2 up:standby 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:41.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:41.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:41.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:41.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:40 smithi045 ceph-mon[36308]: Deploying daemon mds.cephfs.smithi045.hxlkks on smithi045 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:boot 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2 up:standby 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T11:39:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:40 smithi044 ceph-mon[28731]: Deploying daemon mds.cephfs.smithi045.hxlkks on smithi045 2024-04-16T11:39:42.206 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:42 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 -- 172.21.15.44:0/2021587781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fdff0 msgr2=0x7f23f40fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 --2- 172.21.15.44:0/2021587781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fdff0 0x7f23f40fe460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f23e0004940 tx=0x7f23e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 -- 172.21.15.44:0/2021587781 shutdown_connections 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 --2- 172.21.15.44:0/2021587781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fdff0 0x7f23f40fe460 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 --2- 172.21.15.44:0/2021587781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fce80 0x7f23f40fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 -- 172.21.15.44:0/2021587781 >> 172.21.15.44:0/2021587781 conn(0x7f23f40f8580 msgr2=0x7f23f40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 -- 172.21.15.44:0/2021587781 shutdown_connections 2024-04-16T11:39:42.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.289+0000 7f23f9000700 1 -- 172.21.15.44:0/2021587781 wait complete. 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.290+0000 7f23f9000700 1 Processor -- start 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.290+0000 7f23f9000700 1 -- start start 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 0x7f23f4071c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f9000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fdff0 0x7f23f4070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40722a0 con 0x7f23f40fce80 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f9000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4070880 con 0x7f23f40fdff0 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 0x7f23f4071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 0x7f23f4071c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51254/0 (socket says 172.21.15.44:51254) 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f1e00700 1 -- 172.21.15.44:0/2307289378 learned_addr learned my addr 172.21.15.44:0/2307289378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.291+0000 7f23f1400700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fdff0 0x7f23f4070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23f1e00700 1 -- 172.21.15.44:0/2307289378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fdff0 msgr2=0x7f23f4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23f1e00700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fdff0 0x7f23f4070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23f1e00700 1 -- 172.21.15.44:0/2307289378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e0005040 con 0x7f23f40fce80 2024-04-16T11:39:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23f1400700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fdff0 0x7f23f4070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:39:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23f1e00700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 0x7f23f4071c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b790 tx=0x7f23e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e400f770 con 0x7f23f40fce80 2024-04-16T11:39:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23e400fda0 con 0x7f23f40fce80 2024-04-16T11:39:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e400d310 con 0x7f23f40fce80 2024-04-16T11:39:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.292+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f4070b60 con 0x7f23f40fce80 2024-04-16T11:39:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.293+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f4071120 con 0x7f23f40fce80 2024-04-16T11:39:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.294+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f404c710 con 0x7f23f40fce80 2024-04-16T11:39:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.295+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f23e4010420 con 0x7f23f40fce80 2024-04-16T11:39:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.296+0000 7f23ea000700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f23d80692a0 0x7f23d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.296+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f23e40870b0 con 0x7f23f40fce80 2024-04-16T11:39:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.296+0000 7f23f1400700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f23d80692a0 0x7f23d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.297+0000 7f23f1400700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f23d80692a0 0x7f23d806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f23e000b3b0 tx=0x7f23e0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:42.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.301+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f23e4056f10 con 0x7f23f40fce80 2024-04-16T11:39:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:42 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:42.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:42.568+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f23f4105960 con 0x7f23f40fce80 2024-04-16T11:39:43.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.015+0000 7f23ea000700 1 -- 172.21.15.44:0/2307289378 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f23e4056aa0 con 0x7f23f40fce80 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f23d80692a0 msgr2=0x7f23d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f23d80692a0 0x7f23d806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f23e000b3b0 tx=0x7f23e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 msgr2=0x7f23f4071c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 0x7f23f4071c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b790 tx=0x7f23e4009250 comp rx=0 tx=0).stop 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 shutdown_connections 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f23d80692a0 0x7f23d806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23f40fce80 0x7f23f4071c10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 --2- 172.21.15.44:0/2307289378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23f40fdff0 0x7f23f4070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.020+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 >> 172.21.15.44:0/2307289378 conn(0x7f23f40f8580 msgr2=0x7f23f40f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.021+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 shutdown_connections 2024-04-16T11:39:43.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:43.021+0000 7f23f9000700 1 -- 172.21.15.44:0/2307289378 wait complete. 2024-04-16T11:39:43.321 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:43 smithi044 ceph-mon[28731]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T11:39:43.321 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:43 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2307289378' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-16T11:39:43.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:43 smithi045 ceph-mon[36308]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T11:39:43.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:43 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2307289378' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-16T11:39:43.747 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:43.758 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:43.758 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-16T11:39:44.286 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2307289378' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-16T11:39:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2 up:standby 2024-04-16T11:39:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:44 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:44.430 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2307289378' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2 up:standby 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:44.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:44 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:45 smithi045 ceph-mon[36308]: pgmap v227: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T11:39:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:45 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:boot 2024-04-16T11:39:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:45 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:standby 2024-04-16T11:39:45.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:45 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] up:active 2024-04-16T11:39:45.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:45 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:39:45.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:45 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:39:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:45 smithi044 ceph-mon[28731]: pgmap v227: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T11:39:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:45 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:boot 2024-04-16T11:39:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:45 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:standby 2024-04-16T11:39:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:45 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] up:active 2024-04-16T11:39:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:45 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:39:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:45 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:39:46.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 -- 172.21.15.44:0/1241427827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 msgr2=0x7faf800fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:46.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 --2- 172.21.15.44:0/1241427827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf800fd290 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7faf70004940 tx=0x7faf70009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 -- 172.21.15.44:0/1241427827 shutdown_connections 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 --2- 172.21.15.44:0/1241427827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 0x7faf800fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 --2- 172.21.15.44:0/1241427827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf800fd290 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 -- 172.21.15.44:0/1241427827 >> 172.21.15.44:0/1241427827 conn(0x7faf800f8580 msgr2=0x7faf800fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 -- 172.21.15.44:0/1241427827 shutdown_connections 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.436+0000 7faf86400700 1 -- 172.21.15.44:0/1241427827 wait complete. 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.437+0000 7faf86400700 1 Processor -- start 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.437+0000 7faf86400700 1 -- start start 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf86400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf8010fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf86400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 0x7faf80110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf86400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf80110690 con 0x7faf800fce80 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf86400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf80110800 con 0x7faf800fdff0 2024-04-16T11:39:46.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf7f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf8010fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf7f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf8010fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51278/0 (socket says 172.21.15.44:51278) 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf7f400700 1 -- 172.21.15.44:0/3010271862 learned_addr learned my addr 172.21.15.44:0/3010271862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf77e00700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 0x7faf80110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf7f400700 1 -- 172.21.15.44:0/3010271862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 msgr2=0x7faf80110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf7f400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 0x7faf80110090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf7f400700 1 -- 172.21.15.44:0/3010271862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf70005040 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.438+0000 7faf77e00700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 0x7faf80110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.439+0000 7faf7f400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf8010fb50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b3b0 tx=0x7faf70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.439+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf7000e040 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.439+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf70004290 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.439+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf80115200 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.439+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf7001c630 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.439+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf801157c0 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.441+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf80101980 con 0x7faf800fce80 2024-04-16T11:39:46.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.442+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7faf7000bbf0 con 0x7faf800fce80 2024-04-16T11:39:46.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.443+0000 7faf77400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf600692a0 0x7faf6006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:46.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.443+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7faf700885f0 con 0x7faf800fce80 2024-04-16T11:39:46.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.443+0000 7faf77e00700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf600692a0 0x7faf6006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:46.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.444+0000 7faf77e00700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf600692a0 0x7faf6006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faf68007fd0 tx=0x7faf68009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:46.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.448+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7faf700583b0 con 0x7faf800fce80 2024-04-16T11:39:46.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:46.726+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7faf8004c710 con 0x7faf800fce80 2024-04-16T11:39:47.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.017+0000 7faf77400700 1 -- 172.21.15.44:0/3010271862 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7faf70057f40 con 0x7faf800fce80 2024-04-16T11:39:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf600692a0 msgr2=0x7faf6006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf600692a0 0x7faf6006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faf68007fd0 tx=0x7faf68009040 comp rx=0 tx=0).stop 2024-04-16T11:39:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 msgr2=0x7faf8010fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:47.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf8010fb50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b3b0 tx=0x7faf70009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 shutdown_connections 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf600692a0 0x7faf6006b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf800fce80 0x7faf8010fb50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 --2- 172.21.15.44:0/3010271862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf800fdff0 0x7faf80110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 >> 172.21.15.44:0/3010271862 conn(0x7faf800f8580 msgr2=0x7faf800f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 shutdown_connections 2024-04-16T11:39:47.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:47.021+0000 7faf86400700 1 -- 172.21.15.44:0/3010271862 wait complete. 2024-04-16T11:39:47.020 INFO:teuthology.orchestra.run.smithi044.stderr:inline data disabled 2024-04-16T11:39:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:46 smithi044 ceph-mon[28731]: pgmap v228: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 2 op/s 2024-04-16T11:39:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:46 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3010271862' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T11:39:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:39:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:46 smithi045 ceph-mon[36308]: pgmap v228: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 2 op/s 2024-04-16T11:39:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:46 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3010271862' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T11:39:47.847 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:39:47.858 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:39:47.858 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T11:39:48.166 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:48 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3010271862' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T11:39:48.166 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:48 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:39:48.166 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:48.166 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:48 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:48 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3010271862' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T11:39:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:48 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:39:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:48 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:48.672 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:49 smithi044 ceph-mon[28731]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 1 op/s 2024-04-16T11:39:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:49 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:standby 2024-04-16T11:39:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:49 smithi044 ceph-mon[28731]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:standby 2024-04-16T11:39:49.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:49 smithi044 ceph-mon[28731]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:39:49.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:49 smithi045 ceph-mon[36308]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 1 op/s 2024-04-16T11:39:49.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:49 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:standby 2024-04-16T11:39:49.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:49 smithi045 ceph-mon[36308]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:standby 2024-04-16T11:39:49.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:49 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:39:51.280 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:51 smithi044 ceph-mon[28731]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T11:39:51.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:51 smithi045 ceph-mon[36308]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T11:39:51.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 -- 172.21.15.44:0/894143341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c8103420 msgr2=0x7fd8c8103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 --2- 172.21.15.44:0/894143341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c8103420 0x7fd8c8103830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009ce0 tx=0x7fd8b8004d10 comp rx=0 tx=0).stop 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 -- 172.21.15.44:0/894143341 shutdown_connections 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 --2- 172.21.15.44:0/894143341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c81046d0 0x7fd8c8104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 --2- 172.21.15.44:0/894143341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c8103420 0x7fd8c8103830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 -- 172.21.15.44:0/894143341 >> 172.21.15.44:0/894143341 conn(0x7fd8c8072640 msgr2=0x7fd8c8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 -- 172.21.15.44:0/894143341 shutdown_connections 2024-04-16T11:39:51.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.596+0000 7fd8cd600700 1 -- 172.21.15.44:0/894143341 wait complete. 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.597+0000 7fd8cd600700 1 Processor -- start 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.597+0000 7fd8cd600700 1 -- start start 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.597+0000 7fd8cd600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8103420 0x7fd8c8114110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8cd600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 0x7fd8c8114650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8cd600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8114c50 con 0x7fd8c81046d0 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8cd600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8114dc0 con 0x7fd8c8103420 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8be000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 0x7fd8c8114650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:51.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8be000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 0x7fd8c8114650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47544/0 (socket says 172.21.15.44:47544) 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8be000700 1 -- 172.21.15.44:0/4233164852 learned_addr learned my addr 172.21.15.44:0/4233164852 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8c6a00700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8103420 0x7fd8c8114110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8be000700 1 -- 172.21.15.44:0/4233164852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8103420 msgr2=0x7fd8c8114110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8be000700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8103420 0x7fd8c8114110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.598+0000 7fd8be000700 1 -- 172.21.15.44:0/4233164852 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b8009940 con 0x7fd8c81046d0 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.599+0000 7fd8be000700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 0x7fd8c8114650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000c830 tx=0x7fd8b0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.599+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8b000a760 con 0x7fd8c81046d0 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.599+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8b000ad90 con 0x7fd8c81046d0 2024-04-16T11:39:51.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.599+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c8119820 con 0x7fd8c81046d0 2024-04-16T11:39:51.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.599+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8b00113a0 con 0x7fd8c81046d0 2024-04-16T11:39:51.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.599+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c8119de0 con 0x7fd8c81046d0 2024-04-16T11:39:51.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.600+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c804c710 con 0x7fd8c81046d0 2024-04-16T11:39:51.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.601+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd8b0012400 con 0x7fd8c81046d0 2024-04-16T11:39:51.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.602+0000 7fd8bf400700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd8ac0692a0 0x7fd8ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:51.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.602+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd8b0087210 con 0x7fd8c81046d0 2024-04-16T11:39:51.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.603+0000 7fd8c6a00700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd8ac0692a0 0x7fd8ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:51.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.603+0000 7fd8c6a00700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd8ac0692a0 0x7fd8ac06b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8005010 tx=0x7fd8b8004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:51.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.607+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd8b0056fb0 con 0x7fd8c81046d0 2024-04-16T11:39:51.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.878+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd8c811a0b0 con 0x7fd8c81046d0 2024-04-16T11:39:51.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.878+0000 7fd8bf400700 1 -- 172.21.15.44:0/4233164852 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fd8c811a0b0 con 0x7fd8c81046d0 2024-04-16T11:39:51.876 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:39:51.877 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:39:51.877 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:39:51.877 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:39:51.877 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:51.877 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:39:51.877 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:39:51.878 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:39:51.878 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:39:51.878 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:39:51.878 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:39:51.878 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:39:51.878 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:39:51.879 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:39:51.879 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:39:51.879 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:39:51.879 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:39:51.879 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:39:51.879 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:39:51.880 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:39:51.880 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:39:51.880 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:39:51.880 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:39:51.880 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:39:51.881 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:39:51.882 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:39:51.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:51.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:51.882 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:39:51.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:39:51.882 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:39:51.883 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:39:51.883 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:39:51.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd8ac0692a0 msgr2=0x7fd8ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:51.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd8ac0692a0 0x7fd8ac06b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8005010 tx=0x7fd8b8004d10 comp rx=0 tx=0).stop 2024-04-16T11:39:51.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 msgr2=0x7fd8c8114650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 0x7fd8c8114650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000c830 tx=0x7fd8b0004470 comp rx=0 tx=0).stop 2024-04-16T11:39:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 shutdown_connections 2024-04-16T11:39:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd8ac0692a0 0x7fd8ac06b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8103420 0x7fd8c8114110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 --2- 172.21.15.44:0/4233164852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81046d0 0x7fd8c8114650 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.881+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 >> 172.21.15.44:0/4233164852 conn(0x7fd8c8072640 msgr2=0x7fd8c8073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:51.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.882+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 shutdown_connections 2024-04-16T11:39:51.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:51.882+0000 7fd8cd600700 1 -- 172.21.15.44:0/4233164852 wait complete. 2024-04-16T11:39:51.886 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:39:52.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:52 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/4233164852' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:39:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:52 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/4233164852' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:39:52.806 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-16T11:39:53.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:53 smithi045 ceph-mon[36308]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:53.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:53 smithi044 ceph-mon[28731]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:55.152 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:55 smithi044 ceph-mon[28731]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:55.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:55 smithi045 ceph-mon[36308]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:55.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 -- 172.21.15.44:0/1787215208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 msgr2=0x7f0ff40ff9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:55.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1787215208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff40ff9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4004940 tx=0x7f0fe4009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:55.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 -- 172.21.15.44:0/1787215208 shutdown_connections 2024-04-16T11:39:55.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1787215208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ff40fff30 0x7f0ff4102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:55.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1787215208 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff40ff9f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:55.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 -- 172.21.15.44:0/1787215208 >> 172.21.15.44:0/1787215208 conn(0x7f0ff40f7440 msgr2=0x7f0ff40f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:55.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 -- 172.21.15.44:0/1787215208 shutdown_connections 2024-04-16T11:39:55.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.578+0000 7f0ffa200700 1 -- 172.21.15.44:0/1787215208 wait complete. 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.579+0000 7f0ffa200700 1 Processor -- start 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.579+0000 7f0ffa200700 1 -- start start 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.579+0000 7f0ffa200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff410d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.579+0000 7f0ffa200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ff40fff30 0x7f0ff410ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.579+0000 7f0ffa200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff410e4d0 con 0x7f0ff4066f00 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.579+0000 7f0ffa200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff410e640 con 0x7f0ff40fff30 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff410d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff410d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47558/0 (socket says 172.21.15.44:47558) 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 -- 172.21.15.44:0/1634331237 learned_addr learned my addr 172.21.15.44:0/1634331237 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:39:55.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff2a00700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ff40fff30 0x7f0ff410ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:55.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 -- 172.21.15.44:0/1634331237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ff40fff30 msgr2=0x7f0ff410ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:55.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ff40fff30 0x7f0ff410ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:55.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 -- 172.21.15.44:0/1634331237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4005040 con 0x7f0ff4066f00 2024-04-16T11:39:55.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.580+0000 7f0ff3400700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff410d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400b3b0 tx=0x7f0fe4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:55.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.581+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe400e040 con 0x7f0ff4066f00 2024-04-16T11:39:55.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.581+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fe400bbf0 con 0x7f0ff4066f00 2024-04-16T11:39:55.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.581+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff4112ff0 con 0x7f0ff4066f00 2024-04-16T11:39:55.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.581+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff4113520 con 0x7f0ff4066f00 2024-04-16T11:39:55.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.581+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe401c630 con 0x7f0ff4066f00 2024-04-16T11:39:55.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.582+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff404c710 con 0x7f0ff4066f00 2024-04-16T11:39:55.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.583+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0fe400bd60 con 0x7f0ff4066f00 2024-04-16T11:39:55.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.584+0000 7f0feb400700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0fd80692a0 0x7f0fd806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:39:55.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.584+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0fe4087d90 con 0x7f0ff4066f00 2024-04-16T11:39:55.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.584+0000 7f0ff2a00700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0fd80692a0 0x7f0fd806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:39:55.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.585+0000 7f0ff2a00700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0fd80692a0 0x7f0fd806b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0004f40 tx=0x7f0fe000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:39:55.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.592+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0fe4057bf0 con 0x7f0ff4066f00 2024-04-16T11:39:55.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.869+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff41058e0 con 0x7f0ff4066f00 2024-04-16T11:39:55.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.870+0000 7f0feb400700 1 -- 172.21.15.44:0/1634331237 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4683 (secure 0 0 0) 0x7f0fe4057780 con 0x7f0ff4066f00 2024-04-16T11:39:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0fd80692a0 msgr2=0x7f0fd806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0fd80692a0 0x7f0fd806b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0004f40 tx=0x7f0fe000b040 comp rx=0 tx=0).stop 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 msgr2=0x7f0ff410d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff410d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400b3b0 tx=0x7f0fe4009e30 comp rx=0 tx=0).stop 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 shutdown_connections 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f0fd80692a0 0x7f0fd806b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ff4066f00 0x7f0ff410d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 --2- 172.21.15.44:0/1634331237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ff40fff30 0x7f0ff410ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 >> 172.21.15.44:0/1634331237 conn(0x7f0ff40f7440 msgr2=0x7f0ff40f7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.873+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 shutdown_connections 2024-04-16T11:39:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:39:55.874+0000 7f0ffa200700 1 -- 172.21.15.44:0/1634331237 wait complete. 2024-04-16T11:39:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:39:55.887 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:56 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1634331237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T11:39:56.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:56.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:56.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:39:56.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:39:56.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:56.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:39:56.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:56 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1634331237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T11:39:56.755 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-16T11:39:57.428 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:57 smithi045 ceph-mon[36308]: pgmap v233: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:57 smithi044 ceph-mon[28731]: pgmap v233: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:57.740 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:39:59.388 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:59 smithi044 ceph-mon[28731]: pgmap v234: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:59.388 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:59.388 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:39:59 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:59.546 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:59 smithi045 ceph-mon[36308]: pgmap v234: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:39:59.546 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:39:59.546 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:39:59 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 -- 172.21.15.44:0/3511571900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 msgr2=0x7f08200703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 --2- 172.21.15.44:0/3511571900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08200703b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0810004940 tx=0x7f0810009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 -- 172.21.15.44:0/3511571900 shutdown_connections 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 --2- 172.21.15.44:0/3511571900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 0x7f0820070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 --2- 172.21.15.44:0/3511571900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08200703b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 -- 172.21.15.44:0/3511571900 >> 172.21.15.44:0/3511571900 conn(0x7f08200f8530 msgr2=0x7f08200fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 -- 172.21.15.44:0/3511571900 shutdown_connections 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.054+0000 7f0825200700 1 -- 172.21.15.44:0/3511571900 wait complete. 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.055+0000 7f0825200700 1 Processor -- start 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.055+0000 7f0825200700 1 -- start start 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f0825200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 0x7f0820114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:00.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f0825200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08201145c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f0825200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0820114bc0 con 0x7f0820071ea0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f0825200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0820115410 con 0x7f08200708f0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f081d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08201145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f081d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08201145c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47568/0 (socket says 172.21.15.44:47568) 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f081d400700 1 -- 172.21.15.44:0/2181331310 learned_addr learned my addr 172.21.15.44:0/2181331310 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.056+0000 7f081de00700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 0x7f0820114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f081d400700 1 -- 172.21.15.44:0/2181331310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 msgr2=0x7f0820114080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f081d400700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 0x7f0820114080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f081d400700 1 -- 172.21.15.44:0/2181331310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0810005040 con 0x7f0820071ea0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f081de00700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 0x7f0820114080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f081d400700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08201145c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f080c002a50 tx=0x7f080c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f080c007cc0 con 0x7f0820071ea0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f080c00f040 con 0x7f0820071ea0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08201156f0 con 0x7f0820071ea0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f080c009640 con 0x7f0820071ea0 2024-04-16T11:40:00.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.057+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0820115c30 con 0x7f0820071ea0 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.058+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082005f5f0 con 0x7f0820071ea0 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.059+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f080c0097a0 con 0x7f0820071ea0 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.060+0000 7f0816000700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f08040692b0 0x7f080406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.060+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f080c086fd0 con 0x7f0820071ea0 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.061+0000 7f081de00700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f08040692b0 0x7f080406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.061+0000 7f081de00700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f08040692b0 0x7f080406b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f081000b3b0 tx=0x7f0810009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.065+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f080c056d10 con 0x7f0820071ea0 2024-04-16T11:40:00.307 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:00 smithi044 ceph-mon[28731]: overall HEALTH_OK 2024-04-16T11:40:00.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.415+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f082004bed0 con 0x7f0820071ea0 2024-04-16T11:40:00.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.416+0000 7f0816000700 1 -- 172.21.15.44:0/2181331310 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v13) v1 ==== 78+0+99 (secure 0 0 0) 0x7f080c0568a0 con 0x7f0820071ea0 2024-04-16T11:40:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f08040692b0 msgr2=0x7f080406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f08040692b0 0x7f080406b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f081000b3b0 tx=0x7f0810009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 msgr2=0x7f08201145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08201145c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f080c002a50 tx=0x7f080c0046b0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 shutdown_connections 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f08040692b0 0x7f080406b760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08200708f0 0x7f0820114080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 --2- 172.21.15.44:0/2181331310 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0820071ea0 0x7f08201145c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 >> 172.21.15.44:0/2181331310 conn(0x7f08200f8530 msgr2=0x7f08200f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 shutdown_connections 2024-04-16T11:40:00.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:00.419+0000 7f0825200700 1 -- 172.21.15.44:0/2181331310 wait complete. 2024-04-16T11:40:00.432 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:40:00.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:00 smithi045 ceph-mon[36308]: overall HEALTH_OK 2024-04-16T11:40:01.449 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:01 smithi044 ceph-mon[28731]: pgmap v235: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:01.450 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:01 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2181331310' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T11:40:01.450 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:01 smithi045 ceph-mon[36308]: pgmap v235: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:01 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2181331310' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T11:40:01.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:02.027 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-16T11:40:02.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T11:40:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:02 smithi044 ceph-mon[28731]: pgmap v236: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 -- 172.21.15.44:0/3263075226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 msgr2=0x7f84f40fb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 --2- 172.21.15.44:0/3263075226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f40fb310 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009c50 tx=0x7f84e8004d10 comp rx=0 tx=0).stop 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 -- 172.21.15.44:0/3263075226 shutdown_connections 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 --2- 172.21.15.44:0/3263075226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f40fc0f0 0x7f84f40fc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 --2- 172.21.15.44:0/3263075226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f40fb310 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 -- 172.21.15.44:0/3263075226 >> 172.21.15.44:0/3263075226 conn(0x7f84f40f6650 msgr2=0x7f84f40f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:02.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 -- 172.21.15.44:0/3263075226 shutdown_connections 2024-04-16T11:40:02.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.303+0000 7f84fa800640 1 -- 172.21.15.44:0/3263075226 wait complete. 2024-04-16T11:40:02.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.304+0000 7f84fa800640 1 Processor -- start 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.304+0000 7f84fa800640 1 -- start start 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.304+0000 7f84fa800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f410eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.304+0000 7f84fa800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f40fc0f0 0x7f84f410f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.304+0000 7f84fa800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f410f930 con 0x7f84f40faee0 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.304+0000 7f84fa800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f41142e0 con 0x7f84f40fc0f0 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.305+0000 7f84f9400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f410eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.305+0000 7f84f9400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f410eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53292/0 (socket says 172.21.15.44:53292) 2024-04-16T11:40:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.305+0000 7f84f9400640 1 -- 172.21.15.44:0/2575098342 learned_addr learned my addr 172.21.15.44:0/2575098342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.305+0000 7f84f9400640 1 -- 172.21.15.44:0/2575098342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f40fc0f0 msgr2=0x7f84f410f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.305+0000 7f84f9400640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f40fc0f0 0x7f84f410f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.305+0000 7f84f9400640 1 -- 172.21.15.44:0/2575098342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e80098b0 con 0x7f84f40faee0 2024-04-16T11:40:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.306+0000 7f84f9400640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f410eda0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f84e8004a30 tx=0x7f84e8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.306+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84e803a070 con 0x7f84f40faee0 2024-04-16T11:40:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.306+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84e8009710 con 0x7f84f40faee0 2024-04-16T11:40:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.306+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84e803eaf0 con 0x7f84f40faee0 2024-04-16T11:40:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.306+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f4114420 con 0x7f84f40faee0 2024-04-16T11:40:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.306+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f4114990 con 0x7f84f40faee0 2024-04-16T11:40:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.307+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f404bcc0 con 0x7f84f40faee0 2024-04-16T11:40:02.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.308+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f84e8032070 con 0x7f84f40faee0 2024-04-16T11:40:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.310+0000 7f84f1600640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84dc069220 0x7f84dc06b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.310+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f84e80a9340 con 0x7f84f40faee0 2024-04-16T11:40:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.310+0000 7f84f8a00640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84dc069220 0x7f84dc06b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.310+0000 7f84f8a00640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84dc069220 0x7f84dc06b6d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f84f4110250 tx=0x7f84e0009c50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:02.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.316+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f84e80755d0 con 0x7f84f40faee0 2024-04-16T11:40:02.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.518+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84f41009f0 con 0x7f84f40faee0 2024-04-16T11:40:02.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.519+0000 7f84f1600640 1 -- 172.21.15.44:0/2575098342 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14216 (secure 0 0 0) 0x7f84e8078bf0 con 0x7f84f40faee0 2024-04-16T11:40:02.516 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:40:02.516 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":45,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:39:29.520784+0000","last_up_change":"2024-04-16T11:37:49.996185+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T11:39:26.412859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T11:39:27.033592+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6803","nonce":1592689770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6805","nonce":1592689770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6809","nonce":1592689770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1592689770},{"type":"v1","addr":"172.21.15.44:6807","nonce":1592689770}]},"public_addr":"172.21.15.44:6803/1592689770","cluster_addr":"172.21.15.44:6805/1592689770","heartbeat_back_addr":"172.21.15.44:6809/1592689770","heartbeat_front_addr":"172.21.15.44:6807/1592689770","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6811","nonce":412128623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6813","nonce":412128623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6817","nonce":412128623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":412128623},{"type":"v1","addr":"172.21.15.44:6815","nonce":412128623}]},"public_addr":"172.21.15.44:6811/412128623","cluster_addr":"172.21.15.44:6813/412128623","heartbeat_back_addr":"172.21.15.44:6817/412128623","heartbeat_front_addr":"172.21.15.44:6815/412128623","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6819","nonce":2239266596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6821","nonce":2239266596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6825","nonce":2239266596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":2239266596},{"type":"v1","addr":"172.21.15.44:6823","nonce":2239266596}]},"public_addr":"172.21.15.44:6819/2239266596","cluster_addr":"172.21.15.44:6821/2239266596","heartbeat_back_addr":"172.21.15.44:6825/2239266596","heartbeat_front_addr":"172.21.15.44:6823/2239266596","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6801","nonce":3183628521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6803","nonce":3183628521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6807","nonce":3183628521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3183628521},{"type":"v1","addr":"172.21.15.45:6805","nonce":3183628521}]},"public_addr":"172.21.15.45:6801/3183628521","cluster_addr":"172.21.15.45:6803/3183628521","heartbeat_back_addr":"172.21.15.45:6807/3183628521","heartbeat_front_addr":"172.21.15.45:6805/3183628521","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6809","nonce":884231764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6811","nonce":884231764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6815","nonce":884231764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":884231764},{"type":"v1","addr":"172.21.15.45:6813","nonce":884231764}]},"public_addr":"172.21.15.45:6809/884231764","cluster_addr":"172.21.15.45:6811/884231764","heartbeat_back_addr":"172.21.15.45:6815/884231764","heartbeat_front_addr":"172.21.15.45:6813/884231764","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6817","nonce":1526857616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6819","nonce":1526857616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6823","nonce":1526857616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1526857616},{"type":"v1","addr":"172.21.15.45:6821","nonce":1526857616}]},"public_addr":"172.21.15.45:6817/1526857616","cluster_addr":"172.21.15.45:6819/1526857616","heartbeat_back_addr":"172.21.15.45:6823/1526857616","heartbeat_front_addr":"172.21.15.45:6821/1526857616","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.521+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84dc069220 msgr2=0x7f84dc06b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.521+0000 7f84fa800640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84dc069220 0x7f84dc06b6d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f84f4110250 tx=0x7f84e0009c50 comp rx=0 tx=0).stop 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.521+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 msgr2=0x7f84f410eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.521+0000 7f84fa800640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f410eda0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f84e8004a30 tx=0x7f84e8004d10 comp rx=0 tx=0).stop 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 shutdown_connections 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f84dc069220 0x7f84dc06b6d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84f40fc0f0 0x7f84f410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 --2- 172.21.15.44:0/2575098342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f40faee0 0x7f84f410eda0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 >> 172.21.15.44:0/2575098342 conn(0x7f84f40f6650 msgr2=0x7f84f4065ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 shutdown_connections 2024-04-16T11:40:02.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.522+0000 7f84fa800640 1 -- 172.21.15.44:0/2575098342 wait complete. 2024-04-16T11:40:02.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T11:36:12.538436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T11:39:26.412859+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T11:39:27.033592+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-16T11:40:02.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T11:40:02.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:02 smithi045 ceph-mon[36308]: pgmap v236: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 -- 172.21.15.44:0/652307680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c066260 msgr2=0x7f9a8c0fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 --2- 172.21.15.44:0/652307680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c066260 0x7f9a8c0fec10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0048d0 tx=0x7f9a7c009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 -- 172.21.15.44:0/652307680 shutdown_connections 2024-04-16T11:40:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 --2- 172.21.15.44:0/652307680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 --2- 172.21.15.44:0/652307680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c066260 0x7f9a8c0fec10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 -- 172.21.15.44:0/652307680 >> 172.21.15.44:0/652307680 conn(0x7f9a8c0f66e0 msgr2=0x7f9a8c0f8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 -- 172.21.15.44:0/652307680 shutdown_connections 2024-04-16T11:40:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.762+0000 7f9a8be00640 1 -- 172.21.15.44:0/652307680 wait complete. 2024-04-16T11:40:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.763+0000 7f9a8be00640 1 Processor -- start 2024-04-16T11:40:02.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.763+0000 7f9a8be00640 1 -- start start 2024-04-16T11:40:02.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.763+0000 7f9a8be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a8c066260 0x7f9a8c10ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.763+0000 7f9a8be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c10f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.763+0000 7f9a8be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c10f910 con 0x7f9a8c0ff150 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.763+0000 7f9a8be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c114280 con 0x7f9a8c066260 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c10f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c10f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53310/0 (socket says 172.21.15.44:53310) 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 -- 172.21.15.44:0/4066959174 learned_addr learned my addr 172.21.15.44:0/4066959174 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 -- 172.21.15.44:0/4066959174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a8c066260 msgr2=0x7f9a8c10ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8aa00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a8c066260 0x7f9a8c10ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a8c066260 0x7f9a8c10ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 -- 172.21.15.44:0/4066959174 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c005040 con 0x7f9a8c0ff150 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.764+0000 7f9a8a000640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c10f2c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c810 tx=0x7f9a780043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.765+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a7800a820 con 0x7f9a8c0ff150 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.765+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a7800ae50 con 0x7f9a8c0ff150 2024-04-16T11:40:02.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.765+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8c114420 con 0x7f9a8c0ff150 2024-04-16T11:40:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.765+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a780115c0 con 0x7f9a8c0ff150 2024-04-16T11:40:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.765+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c1148f0 con 0x7f9a8c0ff150 2024-04-16T11:40:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.766+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8c04bcc0 con 0x7f9a8c0ff150 2024-04-16T11:40:02.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.767+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9a7800a980 con 0x7f9a8c0ff150 2024-04-16T11:40:02.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.768+0000 7f9a82a00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9a70069220 0x7f9a7006b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:02.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.768+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9a78087750 con 0x7f9a8c0ff150 2024-04-16T11:40:02.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.769+0000 7f9a8aa00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9a70069220 0x7f9a7006b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:02.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.769+0000 7f9a8aa00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9a70069220 0x7f9a7006b6d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:02.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.772+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9a78057470 con 0x7f9a8c0ff150 2024-04-16T11:40:02.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.967+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a8c114bc0 con 0x7f9a8c0ff150 2024-04-16T11:40:02.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.967+0000 7f9a82a00640 1 -- 172.21.15.44:0/4066959174 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a78057000 con 0x7f9a8c0ff150 2024-04-16T11:40:02.964 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-16T11:40:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9a70069220 msgr2=0x7f9a7006b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9a70069220 0x7f9a7006b6d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 msgr2=0x7f9a8c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c10f2c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c810 tx=0x7f9a780043c0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 shutdown_connections 2024-04-16T11:40:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9a70069220 0x7f9a7006b6d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a8c0ff150 0x7f9a8c10f2c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 --2- 172.21.15.44:0/4066959174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a8c066260 0x7f9a8c10ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 >> 172.21.15.44:0/4066959174 conn(0x7f9a8c0f66e0 msgr2=0x7f9a8c0fac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 shutdown_connections 2024-04-16T11:40:02.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:02.970+0000 7f9a8be00640 1 -- 172.21.15.44:0/4066959174 wait complete. 2024-04-16T11:40:02.987 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T11:40:03.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.179+0000 7f906a400640 1 -- 172.21.15.44:0/2122087514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 msgr2=0x7f90640fedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.179+0000 7f906a400640 1 --2- 172.21.15.44:0/2122087514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fedb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f90540048d0 tx=0x7f9054009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.179+0000 7f906a400640 1 -- 172.21.15.44:0/2122087514 shutdown_connections 2024-04-16T11:40:03.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.179+0000 7f906a400640 1 --2- 172.21.15.44:0/2122087514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640ff2f0 0x7f90641016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.179+0000 7f906a400640 1 --2- 172.21.15.44:0/2122087514 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fedb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.180+0000 7f906a400640 1 -- 172.21.15.44:0/2122087514 >> 172.21.15.44:0/2122087514 conn(0x7f90640f6880 msgr2=0x7f90640f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:03.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.180+0000 7f906a400640 1 -- 172.21.15.44:0/2122087514 shutdown_connections 2024-04-16T11:40:03.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.180+0000 7f906a400640 1 -- 172.21.15.44:0/2122087514 wait complete. 2024-04-16T11:40:03.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.180+0000 7f906a400640 1 Processor -- start 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f906a400640 1 -- start start 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f906a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f906a400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640ff2f0 0x7f90640facb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f906a400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90640fcc30 con 0x7f90640662a0 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f906a400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90640fb1f0 con 0x7f90640ff2f0 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f9063400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f9063400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53320/0 (socket says 172.21.15.44:53320) 2024-04-16T11:40:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.181+0000 7f9063400640 1 -- 172.21.15.44:0/2546859486 learned_addr learned my addr 172.21.15.44:0/2546859486 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:03.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.182+0000 7f9062a00640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640ff2f0 0x7f90640facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:03.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.182+0000 7f9063400640 1 -- 172.21.15.44:0/2546859486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640ff2f0 msgr2=0x7f90640facb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.182+0000 7f9063400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640ff2f0 0x7f90640facb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.182+0000 7f9063400640 1 -- 172.21.15.44:0/2546859486 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9054005040 con 0x7f90640662a0 2024-04-16T11:40:03.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.182+0000 7f9063400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fc5e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9054000c00 tx=0x7f9054009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:03.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.187+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90540094c0 con 0x7f90640662a0 2024-04-16T11:40:03.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.187+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9054032e90 con 0x7f90640662a0 2024-04-16T11:40:03.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.187+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90540435f0 con 0x7f90640662a0 2024-04-16T11:40:03.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.187+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90640fb470 con 0x7f90640662a0 2024-04-16T11:40:03.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.187+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90640fb930 con 0x7f90640662a0 2024-04-16T11:40:03.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.188+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906404bcc0 con 0x7f90640662a0 2024-04-16T11:40:03.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.189+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f905402f070 con 0x7f90640662a0 2024-04-16T11:40:03.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.190+0000 7f905b400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9048069220 0x7f904806b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:03.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.191+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f90540a90e0 con 0x7f90640662a0 2024-04-16T11:40:03.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.191+0000 7f9062a00640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9048069220 0x7f904806b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:03.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.191+0000 7f9062a00640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9048069220 0x7f904806b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90640fc390 tx=0x7f904c004c70 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:03.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.195+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9054075330 con 0x7f90640662a0 2024-04-16T11:40:03.330 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:03 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2575098342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:40:03.331 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:03 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/4066959174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:40:03.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.394+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f9064104d10 con 0x7f90640662a0 2024-04-16T11:40:03.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.394+0000 7f905b400640 1 -- 172.21.15.44:0/2546859486 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9054078950 con 0x7f90640662a0 2024-04-16T11:40:03.391 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.396+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9048069220 msgr2=0x7f904806b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9048069220 0x7f904806b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90640fc390 tx=0x7f904c004c70 comp rx=0 tx=0).stop 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 msgr2=0x7f90640fc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fc5e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9054000c00 tx=0x7f9054009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 shutdown_connections 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9048069220 0x7f904806b6d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90640ff2f0 0x7f90640facb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 --2- 172.21.15.44:0/2546859486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90640662a0 0x7f90640fc5e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 >> 172.21.15.44:0/2546859486 conn(0x7f90640f6880 msgr2=0x7f90640f6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:03.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 shutdown_connections 2024-04-16T11:40:03.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.397+0000 7f906a400640 1 -- 172.21.15.44:0/2546859486 wait complete. 2024-04-16T11:40:03.411 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T11:40:03.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:03 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2575098342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:40:03.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:03 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/4066959174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:40:03.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 -- 172.21.15.44:0/4271007181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fd470 msgr2=0x7f25ac0fd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 --2- 172.21.15.44:0/4271007181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fd470 0x7f25ac0fd8e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f25980048d0 tx=0x7f2598009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 -- 172.21.15.44:0/4271007181 shutdown_connections 2024-04-16T11:40:03.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 --2- 172.21.15.44:0/4271007181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fd470 0x7f25ac0fd8e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 --2- 172.21.15.44:0/4271007181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac0fc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 -- 172.21.15.44:0/4271007181 >> 172.21.15.44:0/4271007181 conn(0x7f25ac0f7960 msgr2=0x7f25ac0f9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:03.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 -- 172.21.15.44:0/4271007181 shutdown_connections 2024-04-16T11:40:03.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.603+0000 7f25b2000640 1 -- 172.21.15.44:0/4271007181 wait complete. 2024-04-16T11:40:03.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.604+0000 7f25b2000640 1 Processor -- start 2024-04-16T11:40:03.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.604+0000 7f25b2000640 1 -- start start 2024-04-16T11:40:03.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25b2000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac10ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:03.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25b2000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25ac0fd470 0x7f25ac10f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25b2000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac10faa0 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25b2000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1144a0 con 0x7f25ac0fd470 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25ab400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac10ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25ab400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac10ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53344/0 (socket says 172.21.15.44:53344) 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25ab400640 1 -- 172.21.15.44:0/159739089 learned_addr learned my addr 172.21.15.44:0/159739089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25ab400640 1 -- 172.21.15.44:0/159739089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25ac0fd470 msgr2=0x7f25ac10f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25ab400640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25ac0fd470 0x7f25ac10f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.605+0000 7f25ab400640 1 -- 172.21.15.44:0/159739089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598005040 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.606+0000 7f25ab400640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac10ef10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f259c002a50 tx=0x7f259c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:03.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.606+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259c007c30 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.606+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f259c00f040 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.606+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259c00db50 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.606+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac114640 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.606+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac114b30 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.607+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac04bcc0 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.608+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f259c00dcf0 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.609+0000 7f25a3400640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2590069210 0x7f259006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:03.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.609+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f259c086ac0 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.609+0000 7f25aaa00640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2590069210 0x7f259006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:03.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.610+0000 7f25aaa00640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2590069210 0x7f259006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2598009040 tx=0x7f2598009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:03.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.613+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f259c04d250 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.806+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f25ac104eb0 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.807+0000 7f25a3400640 1 -- 172.21.15.44:0/159739089 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f259c052e10 con 0x7f25ac0fc1a0 2024-04-16T11:40:03.804 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.809+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2590069210 msgr2=0x7f259006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.809+0000 7f25b2000640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2590069210 0x7f259006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2598009040 tx=0x7f2598009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 msgr2=0x7f25ac10ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac10ef10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f259c002a50 tx=0x7f259c004600 comp rx=0 tx=0).stop 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 shutdown_connections 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2590069210 0x7f259006b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25ac0fd470 0x7f25ac10f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 --2- 172.21.15.44:0/159739089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25ac0fc1a0 0x7f25ac10ef10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 >> 172.21.15.44:0/159739089 conn(0x7f25ac0f7960 msgr2=0x7f25ac0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 shutdown_connections 2024-04-16T11:40:03.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:03.810+0000 7f25b2000640 1 -- 172.21.15.44:0/159739089 wait complete. 2024-04-16T11:40:03.824 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T11:40:04.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 -- 172.21.15.44:0/1941465192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 msgr2=0x7f7764101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 --2- 172.21.15.44:0/1941465192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f7764101790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f77500048d0 tx=0x7f7750009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 -- 172.21.15.44:0/1941465192 shutdown_connections 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 --2- 172.21.15.44:0/1941465192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f7764101790 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 --2- 172.21.15.44:0/1941465192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 0x7f77640fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 -- 172.21.15.44:0/1941465192 >> 172.21.15.44:0/1941465192 conn(0x7f77640f6820 msgr2=0x7f77640f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 -- 172.21.15.44:0/1941465192 shutdown_connections 2024-04-16T11:40:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.021+0000 7f7769600640 1 -- 172.21.15.44:0/1941465192 wait complete. 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.022+0000 7f7769600640 1 Processor -- start 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.022+0000 7f7769600640 1 -- start start 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.022+0000 7f7769600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 0x7f776410eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.022+0000 7f7769600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f776410f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.022+0000 7f7769600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776410fa40 con 0x7f77640ff3b0 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.022+0000 7f7769600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7764114440 con 0x7f7764064f80 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f776410f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f776410f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53366/0 (socket says 172.21.15.44:53366) 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 -- 172.21.15.44:0/1379467890 learned_addr learned my addr 172.21.15.44:0/1379467890 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762a00640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 0x7f776410eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 -- 172.21.15.44:0/1379467890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 msgr2=0x7f776410eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 0x7f776410eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 -- 172.21.15.44:0/1379467890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750005040 con 0x7f77640ff3b0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762a00640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 0x7f776410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.023+0000 7f7762000640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f776410f3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7750000c00 tx=0x7f7750009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.024+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77500094c0 con 0x7f77640ff3b0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.024+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7750009620 con 0x7f77640ff3b0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.024+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7764114580 con 0x7f77640ff3b0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.024+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77500435f0 con 0x7f77640ff3b0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.024+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7764114a10 con 0x7f77640ff3b0 2024-04-16T11:40:04.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.025+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7764063f30 con 0x7f77640ff3b0 2024-04-16T11:40:04.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.027+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f775002f070 con 0x7f77640ff3b0 2024-04-16T11:40:04.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.028+0000 7f775aa00640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7748069210 0x7f774806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:04.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.028+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f77500a9220 con 0x7f77640ff3b0 2024-04-16T11:40:04.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.028+0000 7f7762a00640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7748069210 0x7f774806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:04.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.029+0000 7f7762a00640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7748069210 0x7f774806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f77540041d0 tx=0x7f7754009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:04.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.036+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7750075430 con 0x7f77640ff3b0 2024-04-16T11:40:04.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.286+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7764114d40 con 0x7f77640ff3b0 2024-04-16T11:40:04.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.287+0000 7f775aa00640 1 -- 172.21.15.44:0/1379467890 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4683 (secure 0 0 0) 0x7f7750078a50 con 0x7f77640ff3b0 2024-04-16T11:40:04.284 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:40:04.284 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:39:47.009160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14456},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14456":{"gid":14456,"name":"cephfs.smithi044.ufhhlc","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.44:6827/2456403196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2456403196},{"type":"v1","addr":"172.21.15.44:6827","nonce":2456403196}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-16T11:40:04.284 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.289+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7748069210 msgr2=0x7f774806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.289+0000 7f7769600640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7748069210 0x7f774806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f77540041d0 tx=0x7f7754009040 comp rx=0 tx=0).stop 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.289+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 msgr2=0x7f776410f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.289+0000 7f7769600640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f776410f3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7750000c00 tx=0x7f7750009da0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 shutdown_connections 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7748069210 0x7f774806b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77640ff3b0 0x7f776410f3f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 --2- 172.21.15.44:0/1379467890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7764064f80 0x7f776410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 >> 172.21.15.44:0/1379467890 conn(0x7f77640f6820 msgr2=0x7f77640fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 shutdown_connections 2024-04-16T11:40:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:04.290+0000 7f7769600640 1 -- 172.21.15.44:0/1379467890 wait complete. 2024-04-16T11:40:04.304 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 12, 'max_mds': 1, 'flags': 18} 2024-04-16T11:40:04.304 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-16T11:40:04.441 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-16T11:40:04.441 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-16T11:40:04.442 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-16T11:40:04.442 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T11:40:04.442 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:04.442 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-16T11:40:04.442 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T11:40:04.442 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:04.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:04.442 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T11:40:04.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:04 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2546859486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:40:04.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:04 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/159739089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:40:04.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:04 smithi044 ceph-mon[28731]: pgmap v237: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:04.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:04.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-04-16T11:40:04.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:04 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2546859486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:40:04.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:04 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/159739089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:40:04.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:04 smithi045 ceph-mon[36308]: pgmap v237: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-04-16T11:40:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T11:40:04.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:04.621 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-16T11:40:04.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:04.670 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-04-16T11:40:04.797 INFO:teuthology.orchestra.run.smithi044.stderr:Cannot find device "ceph-brx" 2024-04-16T11:40:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T11:40:04.801 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-16T11:40:04.801 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T11:40:05.287 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:05 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1379467890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T11:40:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:05 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:05 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:05.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:05 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1379467890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T11:40:05.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:05 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:05.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:05 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:06.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:40:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:40:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:06 smithi044 ceph-mon[28731]: pgmap v238: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:40:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:40:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:06 smithi045 ceph-mon[36308]: pgmap v238: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 -- 172.21.15.44:0/495364774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fdc70 msgr2=0x7f6bc80fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 --2- 172.21.15.44:0/495364774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fdc70 0x7f6bc80fe040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc004940 tx=0x7f6bbc009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 -- 172.21.15.44:0/495364774 shutdown_connections 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 --2- 172.21.15.44:0/495364774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bc80fe610 0x7f6bc8106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 --2- 172.21.15.44:0/495364774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fdc70 0x7f6bc80fe040 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 -- 172.21.15.44:0/495364774 >> 172.21.15.44:0/495364774 conn(0x7f6bc80f76e0 msgr2=0x7f6bc80f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 -- 172.21.15.44:0/495364774 shutdown_connections 2024-04-16T11:40:07.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.502+0000 7f6bcee00700 1 -- 172.21.15.44:0/495364774 wait complete. 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.503+0000 7f6bcee00700 1 Processor -- start 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.503+0000 7f6bcee00700 1 -- start start 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcee00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bc80fdc70 0x7f6bc8115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcee00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 0x7f6bc8116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcee00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc81166c0 con 0x7f6bc80fe610 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcee00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc810fab0 con 0x7f6bc80fdc70 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcd000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 0x7f6bc8116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcd000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 0x7f6bc8116070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53376/0 (socket says 172.21.15.44:53376) 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.504+0000 7f6bcd000700 1 -- 172.21.15.44:0/3924765126 learned_addr learned my addr 172.21.15.44:0/3924765126 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bcda00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bc80fdc70 0x7f6bc8115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:07.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bcd000700 1 -- 172.21.15.44:0/3924765126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bc80fdc70 msgr2=0x7f6bc8115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:07.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bcd000700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bc80fdc70 0x7f6bc8115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:07.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bcd000700 1 -- 172.21.15.44:0/3924765126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc005040 con 0x7f6bc80fe610 2024-04-16T11:40:07.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bcd000700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 0x7f6bc8116070 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6bb400eb40 tx=0x7f6bb400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:07.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bb4007cc0 con 0x7f6bc80fe610 2024-04-16T11:40:07.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bb400f040 con 0x7f6bc80fe610 2024-04-16T11:40:07.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bb4009620 con 0x7f6bc80fe610 2024-04-16T11:40:07.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.505+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc810fd30 con 0x7f6bc80fe610 2024-04-16T11:40:07.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.506+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc8110270 con 0x7f6bc80fe610 2024-04-16T11:40:07.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.507+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc80668e0 con 0x7f6bc80fe610 2024-04-16T11:40:07.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.508+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6bb4009780 con 0x7f6bc80fe610 2024-04-16T11:40:07.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.509+0000 7f6bc5600700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f6bb00692a0 0x7f6bb006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:07.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.509+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6bb4086e20 con 0x7f6bc80fe610 2024-04-16T11:40:07.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.510+0000 7f6bcda00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f6bb00692a0 0x7f6bb006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:07.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.510+0000 7f6bcda00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f6bb00692a0 0x7f6bb006b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00b0d0 tx=0x7f6bbc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:07.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.514+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6bb404e250 con 0x7f6bc80fe610 2024-04-16T11:40:07.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.781+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6bc8109d50 con 0x7f6bc80fe610 2024-04-16T11:40:07.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.782+0000 7f6bc5600700 1 -- 172.21.15.44:0/3924765126 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+83 (secure 0 0 0) 0x7f6bb4056bf0 con 0x7f6bc80fe610 2024-04-16T11:40:07.781 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T11:40:07.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.785+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f6bb00692a0 msgr2=0x7f6bb006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:07.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.785+0000 7f6bcee00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f6bb00692a0 0x7f6bb006b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00b0d0 tx=0x7f6bbc009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:07.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 msgr2=0x7f6bc8116070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:07.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 0x7f6bc8116070 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6bb400eb40 tx=0x7f6bb400b040 comp rx=0 tx=0).stop 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 shutdown_connections 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f6bb00692a0 0x7f6bb006b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bc80fdc70 0x7f6bc8115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 --2- 172.21.15.44:0/3924765126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6bc80fe610 0x7f6bc8116070 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 >> 172.21.15.44:0/3924765126 conn(0x7f6bc80f76e0 msgr2=0x7f6bc80f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 shutdown_connections 2024-04-16T11:40:07.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:07.786+0000 7f6bcee00700 1 -- 172.21.15.44:0/3924765126 wait complete. 2024-04-16T11:40:08.143 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:07 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3924765126' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T11:40:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:07 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3924765126' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T11:40:08.497 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T11:40:08.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:08.498 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-16T11:40:08.557 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T11:40:08.557 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T11:40:08.557 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T11:40:08.557 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-16T11:40:08.557 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T11:40:08.558 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 41984sec preferred_lft 41984sec 2024-04-16T11:40:08.559 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-16T11:40:08.559 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-16T11:40:08.559 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T11:40:08.559 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:08.559 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-16T11:40:08.560 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T11:40:08.560 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-16T11:40:08.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:08.973 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:08 smithi044 ceph-mon[28731]: pgmap v239: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:09.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:09.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:09.127 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T11:40:09.213 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-16T11:40:09.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:09.217 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-16T11:40:09.280 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-16T11:40:09.280 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T11:40:09.280 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T11:40:09.280 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T11:40:09.280 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T11:40:09.281 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:09.281 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-16T11:40:09.281 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T11:40:09.281 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T11:40:09.281 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T11:40:09.281 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-16T11:40:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:08 smithi045 ceph-mon[36308]: pgmap v239: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:09.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:10.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:10.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:10.123 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-16T11:40:10.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:10.182 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-04-16T11:40:10.238 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:10.238 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-16T11:40:10.238 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:10.238 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-16T11:40:10.238 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-16T11:40:10.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:10.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:10.459 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-16T11:40:10.460 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-16T11:40:10.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:10.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:10.833 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:10.833 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-16T11:40:10.833 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-16T11:40:10.833 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T11:40:10.833 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-16T11:40:10.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:11.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:11.067 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-16T11:40:11.067 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T11:40:11.067 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:11.114 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T11:40:11.115 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T11:40:11.115 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:11.157 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-16T11:40:11.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:11 smithi044 ceph-mon[28731]: pgmap v240: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:11.302 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T11:40:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:11 smithi045 ceph-mon[36308]: pgmap v240: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-16T11:40:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-16T11:40:11.402 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/29550041b19d98def6038010fce4bc4b75eea86cfd79511473e22439253e9780/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/9887053bd8404616f9a8238401496d427b2812a2cd16270f077148ada5a664f4/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/3eccf3ca140eb848904191be50339052d8bd08019c7648217983259abfafcb4c/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/268b89bf92137e756f0ea49ea6cd95a509c880c073e05186cdd971976494f1ec/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/cfbd2b78fe60fe1427898e50d012a8c2375589b34c8887a52c70496bf9dd021f/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/65c77336b6375db0c41a65804a96377be3f1f18e0063b754fbb537d79e490283/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/552ad14d5bfad83b762a155aae6819ff9f997f2443c9a3326c3276c371368043/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/bb47e000b8266d287a58bc2039865609b261cd565846ff5e6cf6b0693dac97bc/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/32582202ae3923b50f36984d52b3a90840213bd6c037f149d4cbdcd5f7e0832d/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/a700e7e8d2557d8cea574a1b54c39e247bd0670c5ba8ba39b1771b249649ff15/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/22bbb5da8d28c478ee3ab797018b88a014131c941c8f692a907d1b0c7d173ea4/merged 2024-04-16T11:40:11.403 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/4ab696865002383557794d21aad40a28ceb7402797efee211fb8bb4543c98816/merged 2024-04-16T11:40:11.404 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-16T11:40:11.404 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:11.404 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:11.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:11.405 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-16T11:40:11.454 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T11:40:11.455 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-16T11:40:11.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-16T11:40:11.540 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-16T11:40:11.583 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/29550041b19d98def6038010fce4bc4b75eea86cfd79511473e22439253e9780/merged 2024-04-16T11:40:11.584 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/9887053bd8404616f9a8238401496d427b2812a2cd16270f077148ada5a664f4/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/3eccf3ca140eb848904191be50339052d8bd08019c7648217983259abfafcb4c/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/268b89bf92137e756f0ea49ea6cd95a509c880c073e05186cdd971976494f1ec/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/cfbd2b78fe60fe1427898e50d012a8c2375589b34c8887a52c70496bf9dd021f/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/65c77336b6375db0c41a65804a96377be3f1f18e0063b754fbb537d79e490283/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/552ad14d5bfad83b762a155aae6819ff9f997f2443c9a3326c3276c371368043/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/bb47e000b8266d287a58bc2039865609b261cd565846ff5e6cf6b0693dac97bc/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/32582202ae3923b50f36984d52b3a90840213bd6c037f149d4cbdcd5f7e0832d/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/a700e7e8d2557d8cea574a1b54c39e247bd0670c5ba8ba39b1771b249649ff15/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/22bbb5da8d28c478ee3ab797018b88a014131c941c8f692a907d1b0c7d173ea4/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/4ab696865002383557794d21aad40a28ceb7402797efee211fb8bb4543c98816/merged 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:11.585 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:11.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:11.586 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-16T11:40:11.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-16T11:40:11.663+0000 7fada952f300 -1 init, newargv = 0x5610cb7d8e80 newargc=16 2024-04-16T11:40:11.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[76454]: starting ceph client 2024-04-16T11:40:11.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:FUSE library version: 2.9.9 2024-04-16T11:40:11.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[76454]: starting fuse 2024-04-16T11:40:11.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stdout:ceph_fuse: starting fuse loop 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:INIT: 7.40 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:flags=0x73fffffb 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:max_readahead=0x00020000 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: INIT: 7.19 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: flags=0x0000043b 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: max_readahead=0x00020000 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: max_write=0x00020000 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: max_background=0 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: congestion_threshold=0 2024-04-16T11:40:11.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 2, success, outsize: 40 2024-04-16T11:40:11.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-16T11:40:11.678+0000 7fada952f300 -1 multithreaded fuse 2024-04-16T11:40:11.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76498 2024-04-16T11:40:11.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 4, success, outsize: 120 2024-04-16T11:40:12.770 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-16T11:40:12.872 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T11:40:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-16T11:40:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-16T11:40:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-16T11:40:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-16T11:40:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-16T11:40:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T11:40:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/29550041b19d98def6038010fce4bc4b75eea86cfd79511473e22439253e9780/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/9887053bd8404616f9a8238401496d427b2812a2cd16270f077148ada5a664f4/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/3eccf3ca140eb848904191be50339052d8bd08019c7648217983259abfafcb4c/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/268b89bf92137e756f0ea49ea6cd95a509c880c073e05186cdd971976494f1ec/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/cfbd2b78fe60fe1427898e50d012a8c2375589b34c8887a52c70496bf9dd021f/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/65c77336b6375db0c41a65804a96377be3f1f18e0063b754fbb537d79e490283/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/552ad14d5bfad83b762a155aae6819ff9f997f2443c9a3326c3276c371368043/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/bb47e000b8266d287a58bc2039865609b261cd565846ff5e6cf6b0693dac97bc/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/32582202ae3923b50f36984d52b3a90840213bd6c037f149d4cbdcd5f7e0832d/merged 2024-04-16T11:40:12.927 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/a700e7e8d2557d8cea574a1b54c39e247bd0670c5ba8ba39b1771b249649ff15/merged 2024-04-16T11:40:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/22bbb5da8d28c478ee3ab797018b88a014131c941c8f692a907d1b0c7d173ea4/merged 2024-04-16T11:40:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/4ab696865002383557794d21aad40a28ceb7402797efee211fb8bb4543c98816/merged 2024-04-16T11:40:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-16T11:40:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T11:40:12.928 INFO:teuthology.orchestra.run.smithi044.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:12.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:12.929 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-16T11:40:12.974 INFO:teuthology.orchestra.run.smithi044.stdout:148 2024-04-16T11:40:12.974 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [148] 2024-04-16T11:40:12.974 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-16T11:40:12.974 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-16T11:40:12.975 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.0")) 2024-04-16T11:40:12.976 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-16T11:40:13.184 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.0.76454.asok 2024-04-16T11:40:13.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:13.198 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:13 smithi044 ceph-mon[28731]: pgmap v241: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:13.202 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76454.asok 2024-04-16T11:40:13.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:13.202 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76454.asok status 2024-04-16T11:40:13.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:13 smithi045 ceph-mon[36308]: pgmap v241: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "45317f95c5587af98fd8eaefff9013a4ba51d2f7", 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "0", 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "76454", 2024-04-16T11:40:13.443 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 14520, 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 14520 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3632682047 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-16T11:40:13.444 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3632682047 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.14520 192.168.0.1:0/3632682047", 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/3632682047", 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 12, 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 45, 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-16T11:40:13.445 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:40:13.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T11:40:13.994 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:15.200 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:15 smithi044 ceph-mon[28731]: pgmap v242: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:15 smithi045 ceph-mon[36308]: pgmap v242: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.171+0000 7f7115000700 1 -- 172.21.15.44:0/1237705145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109dc0 msgr2=0x7f7110103480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.171+0000 7f7115000700 1 --2- 172.21.15.44:0/1237705145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109dc0 0x7f7110103480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f70fc004940 tx=0x7f70fc009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.172+0000 7f7115000700 1 -- 172.21.15.44:0/1237705145 shutdown_connections 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.172+0000 7f7115000700 1 --2- 172.21.15.44:0/1237705145 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109dc0 0x7f7110103480 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.172+0000 7f7115000700 1 --2- 172.21.15.44:0/1237705145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110109420 0x7f71101097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.172+0000 7f7115000700 1 -- 172.21.15.44:0/1237705145 >> 172.21.15.44:0/1237705145 conn(0x7f7110072bd0 msgr2=0x7f7110074fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.172+0000 7f7115000700 1 -- 172.21.15.44:0/1237705145 shutdown_connections 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.172+0000 7f7115000700 1 -- 172.21.15.44:0/1237705145 wait complete. 2024-04-16T11:40:16.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.173+0000 7f7115000700 1 Processor -- start 2024-04-16T11:40:16.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.173+0000 7f7115000700 1 -- start start 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.173+0000 7f7115000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 0x7f7110119f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.173+0000 7f7115000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110109dc0 0x7f711011a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.173+0000 7f7115000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711011aac0 con 0x7f7110109420 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.173+0000 7f7115000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71101140a0 con 0x7f7110109dc0 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 0x7f7110119f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 0x7f7110119f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36840/0 (socket says 172.21.15.44:36840) 2024-04-16T11:40:16.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 -- 172.21.15.44:0/770870774 learned_addr learned my addr 172.21.15.44:0/770870774 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710d400700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110109dc0 0x7f711011a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 -- 172.21.15.44:0/770870774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110109dc0 msgr2=0x7f711011a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110109dc0 0x7f711011a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 -- 172.21.15.44:0/770870774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc005040 con 0x7f7110109420 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.174+0000 7f710de00700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 0x7f7110119f30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f710000eb40 tx=0x7f710000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.175+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7100007cc0 con 0x7f7110109420 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.175+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f710000f040 con 0x7f7110109420 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.175+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7100009620 con 0x7f7110109420 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.175+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7110114320 con 0x7f7110109420 2024-04-16T11:40:16.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.175+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7110114860 con 0x7f7110109420 2024-04-16T11:40:16.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.176+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7110104b70 con 0x7f7110109420 2024-04-16T11:40:16.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.177+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7100009780 con 0x7f7110109420 2024-04-16T11:40:16.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.178+0000 7f7106000700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f70f40692a0 0x7f70f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:16.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.178+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7100086db0 con 0x7f7110109420 2024-04-16T11:40:16.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.179+0000 7f710d400700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f70f40692a0 0x7f70f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:16.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.179+0000 7f710d400700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f70f40692a0 0x7f70f406b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00b0d0 tx=0x7f70fc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.183+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f710004e250 con 0x7f7110109420 2024-04-16T11:40:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:16.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.450+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7110115580 con 0x7f7110109420 2024-04-16T11:40:16.449 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:16.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.451+0000 7f7106000700 1 -- 172.21.15.44:0/770870774 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+83 (secure 0 0 0) 0x7f7100056b80 con 0x7f7110109420 2024-04-16T11:40:16.449 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T11:40:16.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f70f40692a0 msgr2=0x7f70f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:16.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f70f40692a0 0x7f70f406b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00b0d0 tx=0x7f70fc009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:16.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 msgr2=0x7f7110119f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:16.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 0x7f7110119f30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f710000eb40 tx=0x7f710000b040 comp rx=0 tx=0).stop 2024-04-16T11:40:16.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 shutdown_connections 2024-04-16T11:40:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f70f40692a0 0x7f70f406b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7110109420 0x7f7110119f30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 --2- 172.21.15.44:0/770870774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110109dc0 0x7f711011a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.454+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 >> 172.21.15.44:0/770870774 conn(0x7f7110072bd0 msgr2=0x7f7110074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.455+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 shutdown_connections 2024-04-16T11:40:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:16.455+0000 7f7115000700 1 -- 172.21.15.44:0/770870774 wait complete. 2024-04-16T11:40:17.201 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:17 smithi044 ceph-mon[28731]: pgmap v243: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:17.201 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:17 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/770870774' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T11:40:17.266 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T11:40:17.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:17.267 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-04-16T11:40:17.328 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T11:40:17.328 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T11:40:17.328 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T11:40:17.328 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T11:40:17.328 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-04-16T11:40:17.328 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:17.329 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 41973sec preferred_lft 41973sec 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-04-16T11:40:17.330 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-04-16T11:40:17.331 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T11:40:17.331 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T11:40:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:17 smithi045 ceph-mon[36308]: pgmap v243: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:17 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/770870774' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T11:40:17.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:17.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:17.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:17.901 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T11:40:17.977 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-16T11:40:17.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:17.981 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-04-16T11:40:18.044 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-04-16T11:40:18.044 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T11:40:18.044 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T11:40:18.044 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T11:40:18.044 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T11:40:18.045 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:18.045 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T11:40:18.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T11:40:18.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T11:40:18.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T11:40:18.046 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T11:40:18.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:18.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:18.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:18.819 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-04-16T11:40:18.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:18.866 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-04-16T11:40:18.957 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:18.957 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T11:40:18.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:18.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-16T11:40:18.957 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T11:40:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:19.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:19 smithi045 ceph-mon[36308]: pgmap v244: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:19.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:19.152 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-16T11:40:19.152 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:19.152 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T11:40:19.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-16T11:40:19.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T11:40:19.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-16T11:40:19.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-16T11:40:19.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-16T11:40:19.153 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T11:40:19.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:19.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:19 smithi044 ceph-mon[28731]: pgmap v244: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:19.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:19.498 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T11:40:19.498 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-04-16T11:40:19.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-04-16T11:40:19.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T11:40:19.498 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-04-16T11:40:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T11:40:19.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:19.737 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-16T11:40:19.737 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T11:40:19.737 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:19.786 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T11:40:19.786 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T11:40:19.786 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:19.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-04-16T11:40:19.976 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T11:40:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-04-16T11:40:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:20.033 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/74a9b2d1a368d86ea553bcb1ee49f585062e48487978d5e918125754919e4a21/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/0d0e858be2711a556d67de28b08f0ecc253cc83319c98efe6af30bfb91696d8d/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a3002e64d29361bbe01252d1d13b25e9eb0b56ad51f2545c952acf81f99c9da7/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/848ed6edfa7a75b623f3621c32d59b08cca840c31d548dccd79b4137cbed0fa2/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/62676ce2b872715e461e8fa65be419177a9095f19bad34e5e10f2f5dd84e071b/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/2e85d98e5c69046afb6a6ab30ec54531f3404770bbf161c0043e63702e71647c/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/aeb0529c08a898faf205d06783303d2a3c9fd764ce10ed94fa569cb25cbad625/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/8577c2b3a8954cd4e7bd4e5f2ea16dbbddcd7713214d59c8120f4481e50a6db2/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/50d9115c1c727f6d10e67cc63674debac9d5b71d7e21d67b5c7a753cc7140b5e/merged 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:20.034 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:20.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:20.035 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-04-16T11:40:20.088 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T11:40:20.088 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-16T11:40:20.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-04-16T11:40:20.176 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T11:40:20.221 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-04-16T11:40:20.221 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-04-16T11:40:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-04-16T11:40:20.223 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/74a9b2d1a368d86ea553bcb1ee49f585062e48487978d5e918125754919e4a21/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/0d0e858be2711a556d67de28b08f0ecc253cc83319c98efe6af30bfb91696d8d/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a3002e64d29361bbe01252d1d13b25e9eb0b56ad51f2545c952acf81f99c9da7/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/848ed6edfa7a75b623f3621c32d59b08cca840c31d548dccd79b4137cbed0fa2/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/62676ce2b872715e461e8fa65be419177a9095f19bad34e5e10f2f5dd84e071b/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/2e85d98e5c69046afb6a6ab30ec54531f3404770bbf161c0043e63702e71647c/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/aeb0529c08a898faf205d06783303d2a3c9fd764ce10ed94fa569cb25cbad625/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/8577c2b3a8954cd4e7bd4e5f2ea16dbbddcd7713214d59c8120f4481e50a6db2/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/50d9115c1c727f6d10e67cc63674debac9d5b71d7e21d67b5c7a753cc7140b5e/merged 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:20.224 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:20.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:20.225 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-04-16T11:40:20.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:2024-04-16T11:40:20.327+0000 7fefe34c9300 -1 init, newargv = 0x5597c34d2bd0 newargc=16 2024-04-16T11:40:20.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:ceph-fuse[61400]: starting ceph client 2024-04-16T11:40:20.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:FUSE library version: 2.9.9 2024-04-16T11:40:20.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:ceph-fuse[61400]: starting fuse 2024-04-16T11:40:20.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stdout:ceph_fuse: starting fuse loop 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:2024-04-16T11:40:20.340+0000 7fefe34c9300 -1 multithreaded fuse 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:INIT: 7.40 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:flags=0x73fffffb 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:max_readahead=0x00020000 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: INIT: 7.19 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: flags=0x0000043b 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: max_readahead=0x00020000 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: max_write=0x00020000 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: max_background=0 2024-04-16T11:40:20.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: congestion_threshold=0 2024-04-16T11:40:20.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 2, success, outsize: 40 2024-04-16T11:40:20.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 61441 2024-04-16T11:40:20.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 4, success, outsize: 120 2024-04-16T11:40:21.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:21 smithi045 ceph-mon[36308]: pgmap v245: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:21.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-04-16T11:40:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:21 smithi044 ceph-mon[28731]: pgmap v245: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:21.523 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T11:40:21.581 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-04-16T11:40:21.581 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-04-16T11:40:21.581 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-04-16T11:40:21.581 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-04-16T11:40:21.581 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-04-16T11:40:21.582 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/74a9b2d1a368d86ea553bcb1ee49f585062e48487978d5e918125754919e4a21/merged 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/0d0e858be2711a556d67de28b08f0ecc253cc83319c98efe6af30bfb91696d8d/merged 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a3002e64d29361bbe01252d1d13b25e9eb0b56ad51f2545c952acf81f99c9da7/merged 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/848ed6edfa7a75b623f3621c32d59b08cca840c31d548dccd79b4137cbed0fa2/merged 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/62676ce2b872715e461e8fa65be419177a9095f19bad34e5e10f2f5dd84e071b/merged 2024-04-16T11:40:21.583 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/2e85d98e5c69046afb6a6ab30ec54531f3404770bbf161c0043e63702e71647c/merged 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/aeb0529c08a898faf205d06783303d2a3c9fd764ce10ed94fa569cb25cbad625/merged 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/8577c2b3a8954cd4e7bd4e5f2ea16dbbddcd7713214d59c8120f4481e50a6db2/merged 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/50d9115c1c727f6d10e67cc63674debac9d5b71d7e21d67b5c7a753cc7140b5e/merged 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T11:40:21.584 INFO:teuthology.orchestra.run.smithi045.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:21.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:21.585 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-04-16T11:40:21.636 INFO:teuthology.orchestra.run.smithi045.stdout:115 2024-04-16T11:40:21.637 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo stdin-killer -- python3 -c ' 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> import glob 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> import re 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> import os 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> import subprocess 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> def _find_admin_socket(client_name): 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> files = glob.glob(asok_path) 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-16T11:40:21.637 DEBUG:teuthology.orchestra.run.smithi045:> 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> # Given a non-glob path, it better be there 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> if "*" not in asok_path: 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> assert(len(files) == 1) 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> return files[0] 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> for f in files: 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> contents = proc_f.read() 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> if mountpoint in contents: 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> return f 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> print(_find_admin_socket("client.1")) 2024-04-16T11:40:21.638 DEBUG:teuthology.orchestra.run.smithi045:> ' 2024-04-16T11:40:21.821 INFO:teuthology.orchestra.run.smithi045.stdout:/var/run/ceph/ceph-client.1.61400.asok 2024-04-16T11:40:21.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-16T11:40:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T11:40:21.841 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.61400.asok 2024-04-16T11:40:21.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:21.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.61400.asok status 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "metadata": { 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_sha1": "45317f95c5587af98fd8eaefff9013a4ba51d2f7", 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "entity_id": "1", 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": "smithi045", 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-16T11:40:22.082 INFO:teuthology.orchestra.run.smithi045.stdout: "pid": "61400", 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "root": "/" 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_count": 0, 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_pinned_count": 0, 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24317, 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "inst": { 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "name": { 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "client", 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 24317 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2024-04-16T11:40:22.083 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3169505875 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3169505875 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "inst_str": "client.24317 192.168.0.1:0/3169505875", 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "addr_str": "192.168.0.1:0/3169505875", 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "inode_count": 1, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_epoch": 12, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch": 45, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch_barrier": 0, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklisted": false, 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs" 2024-04-16T11:40:22.084 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-16T11:40:22.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:22.098 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-16T11:40:22.098 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:22.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 76756 2024-04-16T11:40:22.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 6, success, outsize: 96 2024-04-16T11:40:22.162 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-16T11:40:22.162 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:22.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:22.163 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T11:40:22.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 8, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76789 2024-04-16T11:40:22.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 8, success, outsize: 120 2024-04-16T11:40:22.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 10, opcode: SETATTR (4), nodeid: 1, insize: 128, pid: 76789 2024-04-16T11:40:22.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 10, success, outsize: 120 2024-04-16T11:40:22.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:22.273 DEBUG:teuthology.orchestra.run.smithi045:> stat --file-system '--printf=%T 2024-04-16T11:40:22.273 DEBUG:teuthology.orchestra.run.smithi045:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:22.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 61532 2024-04-16T11:40:22.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 6, success, outsize: 96 2024-04-16T11:40:22.334 INFO:teuthology.orchestra.run.smithi045.stdout:fuseblk 2024-04-16T11:40:22.334 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:22.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:40:22.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T11:40:22.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 8, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 61565 2024-04-16T11:40:22.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 8, success, outsize: 120 2024-04-16T11:40:22.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 10, opcode: SETATTR (4), nodeid: 1, insize: 128, pid: 61565 2024-04-16T11:40:22.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 10, success, outsize: 120 2024-04-16T11:40:22.443 INFO:teuthology.run_tasks:Running task print... 2024-04-16T11:40:22.454 INFO:teuthology.task.print:**** done client 2024-04-16T11:40:22.454 INFO:teuthology.run_tasks:Running task parallel... 2024-04-16T11:40:22.464 INFO:teuthology.task.parallel:starting parallel... 2024-04-16T11:40:22.464 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-16T11:40:22.465 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T11:40:22.465 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:40:22.465 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-04-16T11:40:23.004 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:23.327 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:23 smithi044 ceph-mon[28731]: pgmap v246: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:23.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:23 smithi045 ceph-mon[36308]: pgmap v246: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:25.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.328+0000 7f22ed400700 1 -- 172.21.15.44:0/4215381237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 msgr2=0x7f22e80fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 --2- 172.21.15.44:0/4215381237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 0x7f22e80fe460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f22d4004940 tx=0x7f22d4009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:25.328 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:25 smithi044 ceph-mon[28731]: pgmap v247: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 -- 172.21.15.44:0/4215381237 shutdown_connections 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 --2- 172.21.15.44:0/4215381237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 0x7f22e80fe460 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 --2- 172.21.15.44:0/4215381237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22e80fce80 0x7f22e80fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 -- 172.21.15.44:0/4215381237 >> 172.21.15.44:0/4215381237 conn(0x7f22e80f8580 msgr2=0x7f22e80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 -- 172.21.15.44:0/4215381237 shutdown_connections 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.329+0000 7f22ed400700 1 -- 172.21.15.44:0/4215381237 wait complete. 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.330+0000 7f22ed400700 1 Processor -- start 2024-04-16T11:40:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.330+0000 7f22ed400700 1 -- start start 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22ed400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22e80fce80 0x7f22e8113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22ed400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 0x7f22e81144d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22ed400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8114ad0 con 0x7f22e80fdff0 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22ed400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8114c40 con 0x7f22e80fce80 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22e5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 0x7f22e81144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22e5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 0x7f22e81144d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41000/0 (socket says 172.21.15.44:41000) 2024-04-16T11:40:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22e5e00700 1 -- 172.21.15.44:0/2176355867 learned_addr learned my addr 172.21.15.44:0/2176355867 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:25.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:25 smithi045 ceph-mon[36308]: pgmap v247: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.331+0000 7f22e6800700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22e80fce80 0x7f22e8113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22e5e00700 1 -- 172.21.15.44:0/2176355867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22e80fce80 msgr2=0x7f22e8113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22e5e00700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22e80fce80 0x7f22e8113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22e5e00700 1 -- 172.21.15.44:0/2176355867 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4005040 con 0x7f22e80fdff0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22e6800700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22e80fce80 0x7f22e8113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22e5e00700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22e80fdff0 0x7f22e81144d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f22d400b3b0 tx=0x7f22d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d400e040 con 0x7f22e80fdff0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22d4004290 con 0x7f22e80fdff0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22ed400700 1 -- 172.21.15.44:0/2176355867 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e81195f0 con 0x7f22e80fdff0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.332+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d401c630 con 0x7f22e80fdff0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.333+0000 7f22ed400700 1 -- 172.21.15.44:0/2176355867 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e8119bb0 con 0x7f22e80fdff0 2024-04-16T11:40:25.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.334+0000 7f22ed400700 1 -- 172.21.15.44:0/2176355867 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e8101980 con 0x7f22e80fdff0 2024-04-16T11:40:25.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.336+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f22d400bbf0 con 0x7f22e80fdff0 2024-04-16T11:40:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.337+0000 7f22dea00700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f22cc0692a0 0x7f22cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.337+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f22d4088690 con 0x7f22e80fdff0 2024-04-16T11:40:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.337+0000 7f22e6800700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f22cc0692a0 0x7f22cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:25.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.338+0000 7f22e6800700 1 --2- 172.21.15.44:0/2176355867 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f22cc0692a0 0x7f22cc06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009fd0 tx=0x7f22d800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:25.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.342+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f22d40583b0 con 0x7f22e80fdff0 2024-04-16T11:40:25.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.561+0000 7f22ed400700 1 -- 172.21.15.44:0/2176355867 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f22e804c710 con 0x7f22e80fdff0 2024-04-16T11:40:25.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:25.562+0000 7f22dea00700 1 -- 172.21.15.44:0/2176355867 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7f22d4057f40 con 0x7f22e80fdff0 2024-04-16T11:40:25.562 INFO:teuthology.orchestra.run.smithi044.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-04-16T11:40:26.411 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T11:40:26.412 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:40:26.412 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-16T11:40:26.899 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:27.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:27 smithi045 ceph-mon[36308]: pgmap v248: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:27.338 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:27 smithi044 ceph-mon[28731]: pgmap v248: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.014+0000 7ff727200700 1 -- 172.21.15.44:0/102748427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720103420 msgr2=0x7ff720103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.014+0000 7ff727200700 1 --2- 172.21.15.44:0/102748427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720103420 0x7ff720103830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff710004940 tx=0x7ff710009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.014+0000 7ff727200700 1 -- 172.21.15.44:0/102748427 shutdown_connections 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.014+0000 7ff727200700 1 --2- 172.21.15.44:0/102748427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7201046d0 0x7ff720104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.014+0000 7ff727200700 1 --2- 172.21.15.44:0/102748427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff720103420 0x7ff720103830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.014+0000 7ff727200700 1 -- 172.21.15.44:0/102748427 >> 172.21.15.44:0/102748427 conn(0x7ff720072640 msgr2=0x7ff720074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:29.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.015+0000 7ff727200700 1 -- 172.21.15.44:0/102748427 shutdown_connections 2024-04-16T11:40:29.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.015+0000 7ff727200700 1 -- 172.21.15.44:0/102748427 wait complete. 2024-04-16T11:40:29.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.016+0000 7ff727200700 1 Processor -- start 2024-04-16T11:40:29.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.016+0000 7ff727200700 1 -- start start 2024-04-16T11:40:29.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.016+0000 7ff727200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff720103420 0x7ff720113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:29.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.016+0000 7ff727200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 0x7ff7201144b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:29.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.016+0000 7ff727200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720114ab0 con 0x7ff7201046d0 2024-04-16T11:40:29.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.016+0000 7ff727200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720114c20 con 0x7ff720103420 2024-04-16T11:40:29.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 0x7ff7201144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:29.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 0x7ff7201144b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41014/0 (socket says 172.21.15.44:41014) 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71f400700 1 -- 172.21.15.44:0/3879065873 learned_addr learned my addr 172.21.15.44:0/3879065873 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71fe00700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff720103420 0x7ff720113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71f400700 1 -- 172.21.15.44:0/3879065873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff720103420 msgr2=0x7ff720113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71f400700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff720103420 0x7ff720113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.017+0000 7ff71f400700 1 -- 172.21.15.44:0/3879065873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff710005040 con 0x7ff7201046d0 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.018+0000 7ff71f400700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 0x7ff7201144b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00c8c0 tx=0x7ff70c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:29.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.018+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70c00a980 con 0x7ff7201046d0 2024-04-16T11:40:29.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.018+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff720119680 con 0x7ff7201046d0 2024-04-16T11:40:29.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.018+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff70c00ce60 con 0x7ff7201046d0 2024-04-16T11:40:29.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.018+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70c0115e0 con 0x7ff7201046d0 2024-04-16T11:40:29.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.018+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff720119c40 con 0x7ff7201046d0 2024-04-16T11:40:29.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.019+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72004c710 con 0x7ff7201046d0 2024-04-16T11:40:29.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.021+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff70c012400 con 0x7ff7201046d0 2024-04-16T11:40:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.022+0000 7ff717e00700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7040692a0 0x7ff70406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.022+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff70c087340 con 0x7ff7201046d0 2024-04-16T11:40:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.022+0000 7ff71fe00700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7040692a0 0x7ff70406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:29.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.023+0000 7ff71fe00700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7040692a0 0x7ff70406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff710004f90 tx=0x7ff710009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:29.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.027+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff70c057060 con 0x7ff7201046d0 2024-04-16T11:40:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:29 smithi044 ceph-mon[28731]: pgmap v249: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-16T11:40:29.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.250+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7ff72010bf10 con 0x7ff7201046d0 2024-04-16T11:40:29.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.251+0000 7ff717e00700 1 -- 172.21.15.44:0/3879065873 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff70c0126b0 con 0x7ff7201046d0 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7040692a0 msgr2=0x7ff70406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7040692a0 0x7ff70406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff710004f90 tx=0x7ff710009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 msgr2=0x7ff7201144b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 0x7ff7201144b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00c8c0 tx=0x7ff70c0044c0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 shutdown_connections 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7040692a0 0x7ff70406b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff720103420 0x7ff720113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 --2- 172.21.15.44:0/3879065873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7201046d0 0x7ff7201144b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 >> 172.21.15.44:0/3879065873 conn(0x7ff720072640 msgr2=0x7ff720073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 shutdown_connections 2024-04-16T11:40:29.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:29.254+0000 7ff727200700 1 -- 172.21.15.44:0/3879065873 wait complete. 2024-04-16T11:40:29.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:29 smithi045 ceph-mon[36308]: pgmap v249: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-16T11:40:30.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-16T11:40:30.590 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:31 smithi045 ceph-mon[36308]: pgmap v250: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:31.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:31 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:31 smithi044 ceph-mon[28731]: pgmap v250: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:32.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 -- 172.21.15.44:0/144461906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740662f0 msgr2=0x7faf740ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 --2- 172.21.15.44:0/144461906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740662f0 0x7faf740ffa30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf64004940 tx=0x7faf64009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 -- 172.21.15.44:0/144461906 shutdown_connections 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 --2- 172.21.15.44:0/144461906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf740fff70 0x7faf74102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 --2- 172.21.15.44:0/144461906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740662f0 0x7faf740ffa30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 -- 172.21.15.44:0/144461906 >> 172.21.15.44:0/144461906 conn(0x7faf740f74c0 msgr2=0x7faf740f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 -- 172.21.15.44:0/144461906 shutdown_connections 2024-04-16T11:40:32.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.979+0000 7faf7b000700 1 -- 172.21.15.44:0/144461906 wait complete. 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.980+0000 7faf7b000700 1 Processor -- start 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.980+0000 7faf7b000700 1 -- start start 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf7b000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf740662f0 0x7faf7410fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf7b000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 0x7faf74110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf7b000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf74110860 con 0x7faf740fff70 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf7b000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf74110fa0 con 0x7faf740662f0 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf73400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 0x7faf74110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:32.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf73400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 0x7faf74110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59138/0 (socket says 172.21.15.44:59138) 2024-04-16T11:40:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf73400700 1 -- 172.21.15.44:0/3112915187 learned_addr learned my addr 172.21.15.44:0/3112915187 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.981+0000 7faf73e00700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf740662f0 0x7faf7410fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf73400700 1 -- 172.21.15.44:0/3112915187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf740662f0 msgr2=0x7faf7410fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf73400700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf740662f0 0x7faf7410fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf73400700 1 -- 172.21.15.44:0/3112915187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf64005040 con 0x7faf740fff70 2024-04-16T11:40:32.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf73400700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 0x7faf74110260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faf6000c8c0 tx=0x7faf600044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf6000a900 con 0x7faf740fff70 2024-04-16T11:40:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf6000ce60 con 0x7faf740fff70 2024-04-16T11:40:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf600113a0 con 0x7faf740fff70 2024-04-16T11:40:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.982+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf74111280 con 0x7faf740fff70 2024-04-16T11:40:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.983+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf74111840 con 0x7faf740fff70 2024-04-16T11:40:32.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.984+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7405f5f0 con 0x7faf740fff70 2024-04-16T11:40:32.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.985+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7faf60012400 con 0x7faf740fff70 2024-04-16T11:40:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.986+0000 7faf6be00700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf58069300 0x7faf5806b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:32.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.986+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7faf60087140 con 0x7faf740fff70 2024-04-16T11:40:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.987+0000 7faf73e00700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf58069300 0x7faf5806b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:32.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.987+0000 7faf73e00700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf58069300 0x7faf5806b7b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faf64004890 tx=0x7faf64009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:32.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:32.991+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7faf60056fa0 con 0x7faf740fff70 2024-04-16T11:40:33.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.209+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7faf7404bed0 con 0x7faf740fff70 2024-04-16T11:40:33.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.209+0000 7faf6be00700 1 -- 172.21.15.44:0/3112915187 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7faf60056b30 con 0x7faf740fff70 2024-04-16T11:40:33.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf58069300 msgr2=0x7faf5806b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf58069300 0x7faf5806b7b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faf64004890 tx=0x7faf64009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 msgr2=0x7faf74110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 0x7faf74110260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faf6000c8c0 tx=0x7faf600044c0 comp rx=0 tx=0).stop 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 shutdown_connections 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faf58069300 0x7faf5806b7b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf740662f0 0x7faf7410fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 --2- 172.21.15.44:0/3112915187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faf740fff70 0x7faf74110260 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.212+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 >> 172.21.15.44:0/3112915187 conn(0x7faf740f74c0 msgr2=0x7faf740fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.213+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 shutdown_connections 2024-04-16T11:40:33.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:33.213+0000 7faf7b000700 1 -- 172.21.15.44:0/3112915187 wait complete. 2024-04-16T11:40:33.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:33 smithi044 ceph-mon[28731]: pgmap v251: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:33.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:33 smithi045 ceph-mon[36308]: pgmap v251: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:40:34.152 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-16T11:40:34.670 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:35 smithi044 ceph-mon[28731]: pgmap v252: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:35.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:35 smithi045 ceph-mon[36308]: pgmap v252: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 -- 172.21.15.44:0/2168518233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad80ff020 msgr2=0x7faad80ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 --2- 172.21.15.44:0/2168518233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad80ff020 0x7faad80ff430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faac8004940 tx=0x7faac8009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 -- 172.21.15.44:0/2168518233 shutdown_connections 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 --2- 172.21.15.44:0/2168518233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad81002d0 0x7faad8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 --2- 172.21.15.44:0/2168518233 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad80ff020 0x7faad80ff430 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 -- 172.21.15.44:0/2168518233 >> 172.21.15.44:0/2168518233 conn(0x7faad80fa950 msgr2=0x7faad80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 -- 172.21.15.44:0/2168518233 shutdown_connections 2024-04-16T11:40:36.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.981+0000 7faadda00700 1 -- 172.21.15.44:0/2168518233 wait complete. 2024-04-16T11:40:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.982+0000 7faadda00700 1 Processor -- start 2024-04-16T11:40:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.982+0000 7faadda00700 1 -- start start 2024-04-16T11:40:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faadda00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad80ff020 0x7faad810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faadda00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 0x7faad81100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:36.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faadda00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad81106b0 con 0x7faad81002d0 2024-04-16T11:40:36.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faadda00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8110820 con 0x7faad80ff020 2024-04-16T11:40:36.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faad6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 0x7faad81100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:36.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faad6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 0x7faad81100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59154/0 (socket says 172.21.15.44:59154) 2024-04-16T11:40:36.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faad6000700 1 -- 172.21.15.44:0/3480122862 learned_addr learned my addr 172.21.15.44:0/3480122862 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:36.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.983+0000 7faad6a00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad80ff020 0x7faad810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faad6000700 1 -- 172.21.15.44:0/3480122862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad80ff020 msgr2=0x7faad810fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faad6000700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad80ff020 0x7faad810fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faad6000700 1 -- 172.21.15.44:0/3480122862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac8005040 con 0x7faad81002d0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faad6000700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 0x7faad81100b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faac4002a50 tx=0x7faac40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac4007cc0 con 0x7faad81002d0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faac400f040 con 0x7faad81002d0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac4009640 con 0x7faad81002d0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad8115280 con 0x7faad81002d0 2024-04-16T11:40:36.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.984+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8115840 con 0x7faad81002d0 2024-04-16T11:40:36.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.986+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad804c710 con 0x7faad81002d0 2024-04-16T11:40:36.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.987+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7faac40097a0 con 0x7faad81002d0 2024-04-16T11:40:36.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.987+0000 7faacea00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faabc0692a0 0x7faabc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:36.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.988+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7faac4087070 con 0x7faad81002d0 2024-04-16T11:40:36.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.988+0000 7faad6a00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faabc0692a0 0x7faabc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:36.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.989+0000 7faad6a00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faabc0692a0 0x7faabc06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faac800b3b0 tx=0x7faac8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:36.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:36.993+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7faac4056d10 con 0x7faad81002d0 2024-04-16T11:40:37.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.210+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7faad8107b10 con 0x7faad81002d0 2024-04-16T11:40:37.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.211+0000 7faacea00700 1 -- 172.21.15.44:0/3480122862 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7faac40568a0 con 0x7faad81002d0 2024-04-16T11:40:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faabc0692a0 msgr2=0x7faabc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faabc0692a0 0x7faabc06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faac800b3b0 tx=0x7faac8009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:37.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 msgr2=0x7faad81100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 0x7faad81100b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faac4002a50 tx=0x7faac40046b0 comp rx=0 tx=0).stop 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 shutdown_connections 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7faabc0692a0 0x7faabc06b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad80ff020 0x7faad810fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 --2- 172.21.15.44:0/3480122862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad81002d0 0x7faad81100b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.214+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 >> 172.21.15.44:0/3480122862 conn(0x7faad80fa950 msgr2=0x7faad80fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.215+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 shutdown_connections 2024-04-16T11:40:37.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:37.215+0000 7faadda00700 1 -- 172.21.15.44:0/3480122862 wait complete. 2024-04-16T11:40:37.230 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:37 smithi044 ceph-mon[28731]: pgmap v253: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:37.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:37 smithi045 ceph-mon[36308]: pgmap v253: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:38.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-16T11:40:38.594 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:39 smithi045 ceph-mon[36308]: pgmap v254: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:39.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:39 smithi044 ceph-mon[28731]: pgmap v254: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T11:40:40.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.721+0000 7f19dd800700 1 -- 172.21.15.44:0/3150177699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 msgr2=0x7f19d8070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:40.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.721+0000 7f19dd800700 1 --2- 172.21.15.44:0/3150177699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8070350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f19c8004940 tx=0x7f19c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:40.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.722+0000 7f19dd800700 1 -- 172.21.15.44:0/3150177699 shutdown_connections 2024-04-16T11:40:40.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.722+0000 7f19dd800700 1 --2- 172.21.15.44:0/3150177699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19d8070890 0x7f19d8070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.722+0000 7f19dd800700 1 --2- 172.21.15.44:0/3150177699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8070350 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.722+0000 7f19dd800700 1 -- 172.21.15.44:0/3150177699 >> 172.21.15.44:0/3150177699 conn(0x7f19d80f8580 msgr2=0x7f19d80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:40.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.722+0000 7f19dd800700 1 -- 172.21.15.44:0/3150177699 shutdown_connections 2024-04-16T11:40:40.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.722+0000 7f19dd800700 1 -- 172.21.15.44:0/3150177699 wait complete. 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.723+0000 7f19dd800700 1 Processor -- start 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.723+0000 7f19dd800700 1 -- start start 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.723+0000 7f19dd800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19d8070890 0x7f19d8113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.723+0000 7f19dd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.723+0000 7f19dd800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8114a20 con 0x7f19d8071e40 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.723+0000 7f19dd800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8114b90 con 0x7f19d8070890 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:40.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8114420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59164/0 (socket says 172.21.15.44:59164) 2024-04-16T11:40:40.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6000700 1 -- 172.21.15.44:0/2056618119 learned_addr learned my addr 172.21.15.44:0/2056618119 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:40.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6a00700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19d8070890 0x7f19d8113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:40.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6000700 1 -- 172.21.15.44:0/2056618119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19d8070890 msgr2=0x7f19d8113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:40.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6000700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19d8070890 0x7f19d8113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.724+0000 7f19d6000700 1 -- 172.21.15.44:0/2056618119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8005040 con 0x7f19d8071e40 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.725+0000 7f19d6000700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8114420 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f19c4002a80 tx=0x7f19c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.725+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c4007cc0 con 0x7f19d8071e40 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.725+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19c400f040 con 0x7f19d8071e40 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.725+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c40096f0 con 0x7f19d8071e40 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.725+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d81195f0 con 0x7f19d8071e40 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.725+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d8119bb0 con 0x7f19d8071e40 2024-04-16T11:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.726+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d8101170 con 0x7f19d8071e40 2024-04-16T11:40:40.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.727+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f19c400d690 con 0x7f19d8071e40 2024-04-16T11:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.728+0000 7f19cea00700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f19bc0692a0 0x7f19bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.728+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f19c4086910 con 0x7f19d8071e40 2024-04-16T11:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.729+0000 7f19d6a00700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f19bc0692a0 0x7f19bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.729+0000 7f19d6a00700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f19bc0692a0 0x7f19bc06b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b3b0 tx=0x7f19c8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:40.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.733+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f19c404d250 con 0x7f19d8071e40 2024-04-16T11:40:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.974+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}) v1 -- 0x7f19d802b3e0 con 0x7f19bc0692a0 2024-04-16T11:40:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.987+0000 7f19cea00700 1 -- 172.21.15.44:0/2056618119 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f19d802b3e0 con 0x7f19bc0692a0 2024-04-16T11:40:40.986 INFO:teuthology.orchestra.run.smithi044.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:40:40.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f19bc0692a0 msgr2=0x7f19bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:40.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f19bc0692a0 0x7f19bc06b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b3b0 tx=0x7f19c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 msgr2=0x7f19d8114420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8114420 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f19c4002a80 tx=0x7f19c40046b0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 shutdown_connections 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f19bc0692a0 0x7f19bc06b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19d8070890 0x7f19d8113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 --2- 172.21.15.44:0/2056618119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8071e40 0x7f19d8114420 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.990+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 >> 172.21.15.44:0/2056618119 conn(0x7f19d80f8580 msgr2=0x7f19d80f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.991+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 shutdown_connections 2024-04-16T11:40:40.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:40.991+0000 7f19dd800700 1 -- 172.21.15.44:0/2056618119 wait complete. 2024-04-16T11:40:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:41 smithi044 ceph-mon[28731]: pgmap v255: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T11:40:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:40:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:40:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:40:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:41 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:41 smithi045 ceph-mon[36308]: pgmap v255: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T11:40:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:40:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:40:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:40:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:41 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:40:41.960 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T11:40:41.961 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:40:41.961 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-16T11:40:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:42 smithi044 ceph-mon[28731]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:42 smithi044 ceph-mon[28731]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:40:42.466 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:40:42.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:42 smithi045 ceph-mon[36308]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:42 smithi045 ceph-mon[36308]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:40:43.456 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:43 smithi044 ceph-mon[28731]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:40:43.456 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:43 smithi044 ceph-mon[28731]: pgmap v256: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:43 smithi045 ceph-mon[36308]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T11:40:43.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:43 smithi045 ceph-mon[36308]: pgmap v256: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 -- 172.21.15.44:0/3571131317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 msgr2=0x7fab8c0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 --2- 172.21.15.44:0/3571131317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c0fd250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fab80004940 tx=0x7fab80009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 -- 172.21.15.44:0/3571131317 shutdown_connections 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 --2- 172.21.15.44:0/3571131317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fe040 0x7fab8c0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 --2- 172.21.15.44:0/3571131317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c0fd250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 -- 172.21.15.44:0/3571131317 >> 172.21.15.44:0/3571131317 conn(0x7fab8c0f8530 msgr2=0x7fab8c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 -- 172.21.15.44:0/3571131317 shutdown_connections 2024-04-16T11:40:44.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.751+0000 7fab94400700 1 -- 172.21.15.44:0/3571131317 wait complete. 2024-04-16T11:40:44.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.752+0000 7fab94400700 1 Processor -- start 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.752+0000 7fab94400700 1 -- start start 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab94400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab94400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fe040 0x7fab8c110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab94400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c110f80 con 0x7fab8c0fce40 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab94400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c1110f0 con 0x7fab8c0fe040 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab91a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab91a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c10fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45116/0 (socket says 172.21.15.44:45116) 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab91a00700 1 -- 172.21.15.44:0/104106802 learned_addr learned my addr 172.21.15.44:0/104106802 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.753+0000 7fab91000700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fe040 0x7fab8c110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab91a00700 1 -- 172.21.15.44:0/104106802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fe040 msgr2=0x7fab8c110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab91a00700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fe040 0x7fab8c110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:44.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab91a00700 1 -- 172.21.15.44:0/104106802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80005040 con 0x7fab8c0fce40 2024-04-16T11:40:44.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab91a00700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c10fd30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab8000b3b0 tx=0x7fab80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:44.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab8000e040 con 0x7fab8c0fce40 2024-04-16T11:40:44.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c111370 con 0x7fab8c0fce40 2024-04-16T11:40:44.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.754+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab8000bcc0 con 0x7fab8c0fce40 2024-04-16T11:40:44.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.755+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab8001c630 con 0x7fab8c0fce40 2024-04-16T11:40:44.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.755+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c111930 con 0x7fab8c0fce40 2024-04-16T11:40:44.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.756+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c1019d0 con 0x7fab8c0fce40 2024-04-16T11:40:44.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.757+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fab800139f0 con 0x7fab8c0fce40 2024-04-16T11:40:44.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.758+0000 7fab89600700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fab740692a0 0x7fab7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:44.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.758+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fab80087fe0 con 0x7fab8c0fce40 2024-04-16T11:40:44.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.758+0000 7fab91000700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fab740692a0 0x7fab7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:44.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.759+0000 7fab91000700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fab740692a0 0x7fab7406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab78006fd0 tx=0x7fab78008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:44.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:44.767+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fab80057d60 con 0x7fab8c0fce40 2024-04-16T11:40:45.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.037+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab8c1059c0 con 0x7fab740692a0 2024-04-16T11:40:45.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.040+0000 7fab89600700 1 -- 172.21.15.44:0/104106802 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fab8c1059c0 con 0x7fab740692a0 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fab740692a0 msgr2=0x7fab7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fab740692a0 0x7fab7406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab78006fd0 tx=0x7fab78008040 comp rx=0 tx=0).stop 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 msgr2=0x7fab8c10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c10fd30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab8000b3b0 tx=0x7fab80009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 shutdown_connections 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fab740692a0 0x7fab7406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab8c0fce40 0x7fab8c10fd30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 --2- 172.21.15.44:0/104106802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fe040 0x7fab8c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.042+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 >> 172.21.15.44:0/104106802 conn(0x7fab8c0f8530 msgr2=0x7fab8c0f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.043+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 shutdown_connections 2024-04-16T11:40:45.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.043+0000 7fab94400700 1 -- 172.21.15.44:0/104106802 wait complete. 2024-04-16T11:40:45.056 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 -- 172.21.15.44:0/758927348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 msgr2=0x7f80a4100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 --2- 172.21.15.44:0/758927348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4100720 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8090004940 tx=0x7f8090009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 -- 172.21.15.44:0/758927348 shutdown_connections 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 --2- 172.21.15.44:0/758927348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4100720 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 --2- 172.21.15.44:0/758927348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80a40ff000 0x7f80a40ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 -- 172.21.15.44:0/758927348 >> 172.21.15.44:0/758927348 conn(0x7f80a40fa950 msgr2=0x7f80a40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 -- 172.21.15.44:0/758927348 shutdown_connections 2024-04-16T11:40:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.216+0000 7f80a8a00700 1 -- 172.21.15.44:0/758927348 wait complete. 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.217+0000 7f80a8a00700 1 Processor -- start 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.217+0000 7f80a8a00700 1 -- start start 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a8a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80a40ff000 0x7f80a410fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a8a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a8a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a4110770 con 0x7f80a41002b0 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a8a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a4110fc0 con 0x7f80a40ff000 2024-04-16T11:40:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45136/0 (socket says 172.21.15.44:45136) 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a1400700 1 -- 172.21.15.44:0/3762625192 learned_addr learned my addr 172.21.15.44:0/3762625192 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.218+0000 7f80a1e00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80a40ff000 0x7f80a410fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f80a1400700 1 -- 172.21.15.44:0/3762625192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80a40ff000 msgr2=0x7f80a410fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f80a1400700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80a40ff000 0x7f80a410fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f80a1400700 1 -- 172.21.15.44:0/3762625192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8090005040 con 0x7f80a41002b0 2024-04-16T11:40:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f80a1400700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4110170 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f809000b3b0 tx=0x7f8090009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f809000e040 con 0x7f80a41002b0 2024-04-16T11:40:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8090004290 con 0x7f80a41002b0 2024-04-16T11:40:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a4111240 con 0x7f80a41002b0 2024-04-16T11:40:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.219+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f809001c3f0 con 0x7f80a41002b0 2024-04-16T11:40:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.220+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a4111780 con 0x7f80a41002b0 2024-04-16T11:40:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.221+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a405f5f0 con 0x7f80a41002b0 2024-04-16T11:40:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.222+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f809000bbf0 con 0x7f80a41002b0 2024-04-16T11:40:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.223+0000 7f809a000700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f80880692b0 0x7f808806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.223+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8090088ed0 con 0x7f80a41002b0 2024-04-16T11:40:45.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.224+0000 7f80a1e00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f80880692b0 0x7f808806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:45.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.225+0000 7f80a1e00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f80880692b0 0x7f808806b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f80940041d0 tx=0x7f80940043c0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:45.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.228+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8090058d30 con 0x7f80a41002b0 2024-04-16T11:40:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:45 smithi044 ceph-mon[28731]: pgmap v257: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:45.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:45 smithi045 ceph-mon[36308]: pgmap v257: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:45.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.487+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f80a4111bc0 con 0x7f80880692b0 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.488+0000 7f809a000700 1 -- 172.21.15.44:0/3762625192 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f80a4111bc0 con 0x7f80880692b0 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.491+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f80880692b0 msgr2=0x7f808806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.491+0000 7f80a8a00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f80880692b0 0x7f808806b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f80940041d0 tx=0x7f80940043c0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.491+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 msgr2=0x7f80a4110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.491+0000 7f80a8a00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4110170 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f809000b3b0 tx=0x7f8090009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 shutdown_connections 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f80880692b0 0x7f808806b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80a40ff000 0x7f80a410fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 --2- 172.21.15.44:0/3762625192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80a41002b0 0x7f80a4110170 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 >> 172.21.15.44:0/3762625192 conn(0x7f80a40fa950 msgr2=0x7f80a40fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 shutdown_connections 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.492+0000 7f80a8a00700 1 -- 172.21.15.44:0/3762625192 wait complete. 2024-04-16T11:40:45.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 -- 172.21.15.44:0/488651679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880662f0 msgr2=0x7f59880ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 --2- 172.21.15.44:0/488651679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880662f0 0x7f59880ff800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5978004940 tx=0x7f5978009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 -- 172.21.15.44:0/488651679 shutdown_connections 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 --2- 172.21.15.44:0/488651679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59880ffd40 0x7f5988102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 --2- 172.21.15.44:0/488651679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880662f0 0x7f59880ff800 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 -- 172.21.15.44:0/488651679 >> 172.21.15.44:0/488651679 conn(0x7f59880f7270 msgr2=0x7f59880f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 -- 172.21.15.44:0/488651679 shutdown_connections 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.654+0000 7f5987200700 1 -- 172.21.15.44:0/488651679 wait complete. 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.655+0000 7f5987200700 1 Processor -- start 2024-04-16T11:40:45.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5987200700 1 -- start start 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5987200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59880662f0 0x7f59880fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5987200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 0x7f59880fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5987200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59880fbcb0 con 0x7f59880ffd40 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5987200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59880fbe20 con 0x7f59880662f0 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5985400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 0x7f59880fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5985400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 0x7f59880fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45146/0 (socket says 172.21.15.44:45146) 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.656+0000 7f5985400700 1 -- 172.21.15.44:0/134650881 learned_addr learned my addr 172.21.15.44:0/134650881 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.657+0000 7f5985e00700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59880662f0 0x7f59880fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.657+0000 7f5985400700 1 -- 172.21.15.44:0/134650881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59880662f0 msgr2=0x7f59880fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.657+0000 7f5985400700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59880662f0 0x7f59880fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.657+0000 7f5985400700 1 -- 172.21.15.44:0/134650881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978005040 con 0x7f59880ffd40 2024-04-16T11:40:45.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.657+0000 7f5985400700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 0x7f59880fb700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5974002a50 tx=0x7f59740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.658+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5974007cc0 con 0x7f59880ffd40 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.658+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f597400f040 con 0x7f59880ffd40 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.658+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5974009640 con 0x7f59880ffd40 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.658+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59880fc100 con 0x7f59880ffd40 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.658+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59881158d0 con 0x7f59880ffd40 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.660+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598805f5f0 con 0x7f59880ffd40 2024-04-16T11:40:45.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.660+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f59740097a0 con 0x7f59880ffd40 2024-04-16T11:40:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.662+0000 7f597e000700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f596c0692b0 0x7f596c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.662+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5974086ec0 con 0x7f59880ffd40 2024-04-16T11:40:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.662+0000 7f5985e00700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f596c0692b0 0x7f596c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:45.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.663+0000 7f5985e00700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f596c0692b0 0x7f596c06b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5978000c00 tx=0x7f5978009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:45.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.667+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f59740531f0 con 0x7f59880ffd40 2024-04-16T11:40:45.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.932+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f59880f8db0 con 0x7f596c0692b0 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.980+0000 7f597e000700 1 -- 172.21.15.44:0/134650881 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f59880f8db0 con 0x7f596c0692b0 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (6m) 41s ago 8m 13.5M - 0.25.0 c8568f914cd2 8136acabd663 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (8m) 41s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (7m) 47s ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (6m) 41s ago 7m 75.2M - 9.4.7 954c08fa6188 afde3dcbfd10 2024-04-16T11:40:45.979 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (66s) 41s ago 66s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (74s) 41s ago 73s 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (62s) 47s ago 62s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (70s) 47s ago 70s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:9283 running (10m) 41s ago 10m 483M - 17.2.7-847-g45317f95 b333581ccdf5 53edec877549 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (7m) 47s ago 7m 432M - 17.2.7-847-g45317f95 b333581ccdf5 f6be9ec13da1 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (10m) 41s ago 10m 43.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (6m) 47s ago 6m 34.6M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (8m) 41s ago 8m 7746k - 1.5.0 0da6a335fe13 dbe9480d7dd0 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (7m) 47s ago 7m 7486k - 1.5.0 0da6a335fe13 e37d89867e75 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (5m) 41s ago 5m 44.5M 4290M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (5m) 41s ago 5m 46.9M 4290M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (4m) 41s ago 4m 43.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (4m) 47s ago 4m 42.9M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (3m) 47s ago 3m 40.6M 5314M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:40:45.980 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (3m) 47s ago 3m 42.6M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:40:45.981 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (6m) 41s ago 7m 44.5M - 2.43.0 a07b618ecd1d 99d239fdb386 2024-04-16T11:40:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f596c0692b0 msgr2=0x7f596c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f596c0692b0 0x7f596c06b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5978000c00 tx=0x7f5978009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:45.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 msgr2=0x7f59880fb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 0x7f59880fb700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5974002a50 tx=0x7f59740046b0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 shutdown_connections 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f596c0692b0 0x7f596c06b760 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5978000c00 tx=0x7f5978009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59880662f0 0x7f59880fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 --2- 172.21.15.44:0/134650881 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59880ffd40 0x7f59880fb700 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.984+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 >> 172.21.15.44:0/134650881 conn(0x7f59880f7270 msgr2=0x7f59880f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.985+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 shutdown_connections 2024-04-16T11:40:45.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:45.985+0000 7f5987200700 1 -- 172.21.15.44:0/134650881 wait complete. 2024-04-16T11:40:46.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 -- 172.21.15.44:0/2154972719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fcc30 msgr2=0x7efe800fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 --2- 172.21.15.44:0/2154972719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fcc30 0x7efe800fd040 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efe70004940 tx=0x7efe70009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 -- 172.21.15.44:0/2154972719 shutdown_connections 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 --2- 172.21.15.44:0/2154972719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe800fdee0 0x7efe800fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 --2- 172.21.15.44:0/2154972719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fcc30 0x7efe800fd040 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 -- 172.21.15.44:0/2154972719 >> 172.21.15.44:0/2154972719 conn(0x7efe800f83d0 msgr2=0x7efe800fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 -- 172.21.15.44:0/2154972719 shutdown_connections 2024-04-16T11:40:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.165+0000 7efe7f200700 1 -- 172.21.15.44:0/2154972719 wait complete. 2024-04-16T11:40:46.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.166+0000 7efe7f200700 1 Processor -- start 2024-04-16T11:40:46.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.166+0000 7efe7f200700 1 -- start start 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7f200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe800fcc30 0x7efe8010d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 0x7efe8010dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8010e390 con 0x7efe800fdee0 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7f200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8010ebe0 con 0x7efe800fcc30 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 0x7efe8010dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 0x7efe8010dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45162/0 (socket says 172.21.15.44:45162) 2024-04-16T11:40:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.167+0000 7efe7d400700 1 -- 172.21.15.44:0/2422082106 learned_addr learned my addr 172.21.15.44:0/2422082106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.168+0000 7efe7de00700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe800fcc30 0x7efe8010d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.168+0000 7efe7d400700 1 -- 172.21.15.44:0/2422082106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe800fcc30 msgr2=0x7efe8010d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.168+0000 7efe7d400700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe800fcc30 0x7efe8010d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.168+0000 7efe7d400700 1 -- 172.21.15.44:0/2422082106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70005040 con 0x7efe800fdee0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.168+0000 7efe7d400700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 0x7efe8010dd90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00c8c0 tx=0x7efe6c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.169+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe6c00a980 con 0x7efe800fdee0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.169+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe6c00ce60 con 0x7efe800fdee0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.169+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8010eec0 con 0x7efe800fdee0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.169+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe6c0115e0 con 0x7efe800fdee0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.169+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8010f400 con 0x7efe800fdee0 2024-04-16T11:40:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.170+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8005f5f0 con 0x7efe800fdee0 2024-04-16T11:40:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.172+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7efe6c012400 con 0x7efe800fdee0 2024-04-16T11:40:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.173+0000 7efe76000700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7efe640692b0 0x7efe6406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.173+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7efe6c0871c0 con 0x7efe800fdee0 2024-04-16T11:40:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.173+0000 7efe7de00700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7efe640692b0 0x7efe6406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:46.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.174+0000 7efe7de00700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7efe640692b0 0x7efe6406b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efe7000b3b0 tx=0x7efe70009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:46.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.178+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7efe6c057020 con 0x7efe800fdee0 2024-04-16T11:40:46.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:46 smithi044 ceph-mon[28731]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:46.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.542+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efe8010f050 con 0x7efe800fdee0 2024-04-16T11:40:46.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.544+0000 7efe76000700 1 -- 172.21.15.44:0/2422082106 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7efe8010f050 con 0x7efe800fdee0 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:40:46.543 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:40:46.544 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:40:46.544 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:40:46.544 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T11:40:46.544 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:40:46.544 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:40:46.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7efe640692b0 msgr2=0x7efe6406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:46.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7efe640692b0 0x7efe6406b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efe7000b3b0 tx=0x7efe70009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:46.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 msgr2=0x7efe8010dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:46.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 0x7efe8010dd90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00c8c0 tx=0x7efe6c0044c0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 shutdown_connections 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7efe640692b0 0x7efe6406b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe800fcc30 0x7efe8010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 --2- 172.21.15.44:0/2422082106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efe800fdee0 0x7efe8010dd90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.547+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 >> 172.21.15.44:0/2422082106 conn(0x7efe800f83d0 msgr2=0x7efe800f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.548+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 shutdown_connections 2024-04-16T11:40:46.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.548+0000 7efe7f200700 1 -- 172.21.15.44:0/2422082106 wait complete. 2024-04-16T11:40:46.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:46 smithi045 ceph-mon[36308]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:46.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:40:46.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 -- 172.21.15.44:0/1651922144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc066290 msgr2=0x7fb7fc101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 --2- 172.21.15.44:0/1651922144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc066290 0x7fb7fc101c60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec004940 tx=0x7fb7ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 -- 172.21.15.44:0/1651922144 shutdown_connections 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 --2- 172.21.15.44:0/1651922144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 --2- 172.21.15.44:0/1651922144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc066290 0x7fb7fc101c60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 -- 172.21.15.44:0/1651922144 >> 172.21.15.44:0/1651922144 conn(0x7fb7fc0f7270 msgr2=0x7fb7fc0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 -- 172.21.15.44:0/1651922144 shutdown_connections 2024-04-16T11:40:46.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.722+0000 7fb801200700 1 -- 172.21.15.44:0/1651922144 wait complete. 2024-04-16T11:40:46.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.723+0000 7fb801200700 1 Processor -- start 2024-04-16T11:40:46.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.723+0000 7fb801200700 1 -- start start 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb801200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc066290 0x7fb7fc10fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb801200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc10fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb801200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc1105f0 con 0x7fb7fc1021a0 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb801200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc110e40 con 0x7fb7fc066290 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb7faa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc10fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb7faa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc10fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45190/0 (socket says 172.21.15.44:45190) 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb7faa00700 1 -- 172.21.15.44:0/220384984 learned_addr learned my addr 172.21.15.44:0/220384984 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.724+0000 7fb7fb400700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc066290 0x7fb7fc10fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb7faa00700 1 -- 172.21.15.44:0/220384984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc066290 msgr2=0x7fb7fc10fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb7faa00700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc066290 0x7fb7fc10fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb7faa00700 1 -- 172.21.15.44:0/220384984 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec005040 con 0x7fb7fc1021a0 2024-04-16T11:40:46.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb7faa00700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc10fff0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c8c0 tx=0x7fb7e80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:46.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7e800a760 con 0x7fb7fc1021a0 2024-04-16T11:40:46.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7e800ad90 con 0x7fb7fc1021a0 2024-04-16T11:40:46.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.725+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc111120 con 0x7fb7fc1021a0 2024-04-16T11:40:46.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.726+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7e80113a0 con 0x7fb7fc1021a0 2024-04-16T11:40:46.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.726+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc111660 con 0x7fb7fc1021a0 2024-04-16T11:40:46.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.727+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc05f5f0 con 0x7fb7fc1021a0 2024-04-16T11:40:46.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.729+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb7e8012400 con 0x7fb7fc1021a0 2024-04-16T11:40:46.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.730+0000 7fb7f3400700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb7e00692b0 0x7fb7e006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:46.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.730+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb7e8087bf0 con 0x7fb7fc1021a0 2024-04-16T11:40:46.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.731+0000 7fb7fb400700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb7e00692b0 0x7fb7e006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:46.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.731+0000 7fb7fb400700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb7e00692b0 0x7fb7e006b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec000c00 tx=0x7fb7ec009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:46.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:46.734+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb7e8057a50 con 0x7fb7fc1021a0 2024-04-16T11:40:47.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.006+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb7fc04bed0 con 0x7fb7fc1021a0 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.007+0000 7fb7f3400700 1 -- 172.21.15.44:0/220384984 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fb7e80575e0 con 0x7fb7fc1021a0 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:40:47.005 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:40:47.006 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:40:47.007 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:40:47.008 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:40:47.008 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:40:47.008 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:40:47.008 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.009+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb7e00692b0 msgr2=0x7fb7e006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.009+0000 7fb801200700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb7e00692b0 0x7fb7e006b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec000c00 tx=0x7fb7ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.009+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 msgr2=0x7fb7fc10fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.009+0000 7fb801200700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc10fff0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c8c0 tx=0x7fb7e80044c0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 shutdown_connections 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb7e00692b0 0x7fb7e006b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc066290 0x7fb7fc10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 --2- 172.21.15.44:0/220384984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7fc1021a0 0x7fb7fc10fff0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 >> 172.21.15.44:0/220384984 conn(0x7fb7fc0f7270 msgr2=0x7fb7fc0f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 shutdown_connections 2024-04-16T11:40:47.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.010+0000 7fb801200700 1 -- 172.21.15.44:0/220384984 wait complete. 2024-04-16T11:40:47.010 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.168+0000 7f27bd800700 1 -- 172.21.15.44:0/1905630953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fcc30 msgr2=0x7f27b80fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.168+0000 7f27bd800700 1 --2- 172.21.15.44:0/1905630953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fcc30 0x7f27b80fd040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f27ac004940 tx=0x7f27ac009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.168+0000 7f27bd800700 1 -- 172.21.15.44:0/1905630953 shutdown_connections 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.168+0000 7f27bd800700 1 --2- 172.21.15.44:0/1905630953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b80fdee0 0x7f27b80fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.169+0000 7f27bd800700 1 --2- 172.21.15.44:0/1905630953 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fcc30 0x7f27b80fd040 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.169+0000 7f27bd800700 1 -- 172.21.15.44:0/1905630953 >> 172.21.15.44:0/1905630953 conn(0x7f27b80f83d0 msgr2=0x7f27b80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:47.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.169+0000 7f27bd800700 1 -- 172.21.15.44:0/1905630953 shutdown_connections 2024-04-16T11:40:47.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.169+0000 7f27bd800700 1 -- 172.21.15.44:0/1905630953 wait complete. 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.170+0000 7f27bd800700 1 Processor -- start 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.170+0000 7f27bd800700 1 -- start start 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.170+0000 7f27bd800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b80fcc30 0x7f27b810fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.170+0000 7f27bd800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 0x7f27b8110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.170+0000 7f27bd800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b8110610 con 0x7f27b80fdee0 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.170+0000 7f27bd800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b8110e60 con 0x7f27b80fcc30 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 0x7f27b8110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 0x7f27b8110010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45214/0 (socket says 172.21.15.44:45214) 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7400700 1 -- 172.21.15.44:0/1629140319 learned_addr learned my addr 172.21.15.44:0/1629140319 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:47.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7e00700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b80fcc30 0x7f27b810fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:47.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7400700 1 -- 172.21.15.44:0/1629140319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b80fcc30 msgr2=0x7f27b810fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7400700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b80fcc30 0x7f27b810fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.171+0000 7f27b7400700 1 -- 172.21.15.44:0/1629140319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac005040 con 0x7f27b80fdee0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.172+0000 7f27b7400700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 0x7f27b8110010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f27a4002a50 tx=0x7f27a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.172+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27a4007cc0 con 0x7f27b80fdee0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.172+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27a400f040 con 0x7f27b80fdee0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.172+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27a4009640 con 0x7f27b80fdee0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.172+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b8111140 con 0x7f27b80fdee0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.172+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b8111680 con 0x7f27b80fdee0 2024-04-16T11:40:47.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.173+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b8101870 con 0x7f27b80fdee0 2024-04-16T11:40:47.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.175+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f27a40097a0 con 0x7f27b80fdee0 2024-04-16T11:40:47.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.175+0000 7f27abe00700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f27a00692b0 0x7f27a006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:47.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.176+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f27a40868b0 con 0x7f27b80fdee0 2024-04-16T11:40:47.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.176+0000 7f27b7e00700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f27a00692b0 0x7f27a006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:47.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.177+0000 7f27b7e00700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f27a00692b0 0x7f27a006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00b3b0 tx=0x7f27ac009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:47.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.182+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f27a404d250 con 0x7f27b80fdee0 2024-04-16T11:40:47.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.421+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27b8111960 con 0x7f27a00692b0 2024-04-16T11:40:47.420 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:47 smithi044 ceph-mon[28731]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:47.420 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:47 smithi044 ceph-mon[28731]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:47.420 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:47 smithi044 ceph-mon[28731]: pgmap v258: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:47.420 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:47 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2422082106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:40:47.420 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:47 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/220384984' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:40:47.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.423+0000 7f27abe00700 1 -- 172.21.15.44:0/1629140319 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f27b8111960 con 0x7f27a00692b0 2024-04-16T11:40:47.421 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:40:47.421 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", 2024-04-16T11:40:47.421 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:40:47.421 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:40:47.422 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:40:47.422 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "", 2024-04-16T11:40:47.422 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 image", 2024-04-16T11:40:47.422 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:40:47.422 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:40:47.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.426+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f27a00692b0 msgr2=0x7f27a006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.426+0000 7f27bd800700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f27a00692b0 0x7f27a006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00b3b0 tx=0x7f27ac009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:47.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.426+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 msgr2=0x7f27b8110010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.426+0000 7f27bd800700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 0x7f27b8110010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f27a4002a50 tx=0x7f27a40046b0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 shutdown_connections 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f27a00692b0 0x7f27a006b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f27b80fcc30 0x7f27b810fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 --2- 172.21.15.44:0/1629140319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27b80fdee0 0x7f27b8110010 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 >> 172.21.15.44:0/1629140319 conn(0x7f27b80f83d0 msgr2=0x7f27b80f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:47.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 shutdown_connections 2024-04-16T11:40:47.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.427+0000 7f27bd800700 1 -- 172.21.15.44:0/1629140319 wait complete. 2024-04-16T11:40:47.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:47 smithi045 ceph-mon[36308]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:47.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:47 smithi045 ceph-mon[36308]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:47.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:47 smithi045 ceph-mon[36308]: pgmap v258: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:47.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:47 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2422082106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:40:47.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:47 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/220384984' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:40:47.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 -- 172.21.15.44:0/2501532934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 msgr2=0x7fec3c0fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 --2- 172.21.15.44:0/2501532934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c0fd040 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fec2c004940 tx=0x7fec2c009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:47.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 -- 172.21.15.44:0/2501532934 shutdown_connections 2024-04-16T11:40:47.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 --2- 172.21.15.44:0/2501532934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 0x7fec3c0fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 --2- 172.21.15.44:0/2501532934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c0fd040 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 -- 172.21.15.44:0/2501532934 >> 172.21.15.44:0/2501532934 conn(0x7fec3c0f83d0 msgr2=0x7fec3c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:47.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 -- 172.21.15.44:0/2501532934 shutdown_connections 2024-04-16T11:40:47.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.602+0000 7fec3be00700 1 -- 172.21.15.44:0/2501532934 wait complete. 2024-04-16T11:40:47.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.603+0000 7fec3be00700 1 Processor -- start 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.603+0000 7fec3be00700 1 -- start start 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.603+0000 7fec3be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c10fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.603+0000 7fec3be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 0x7fec3c10ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.603+0000 7fec3be00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c110610 con 0x7fec3c0fcc30 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.603+0000 7fec3be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c110e00 con 0x7fec3c0fdee0 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c10fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c10fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45238/0 (socket says 172.21.15.44:45238) 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3aa00700 1 -- 172.21.15.44:0/2814141333 learned_addr learned my addr 172.21.15.44:0/2814141333 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3a000700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 0x7fec3c10ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3aa00700 1 -- 172.21.15.44:0/2814141333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 msgr2=0x7fec3c10ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3aa00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 0x7fec3c10ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3aa00700 1 -- 172.21.15.44:0/2814141333 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c005040 con 0x7fec3c0fcc30 2024-04-16T11:40:47.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.604+0000 7fec3a000700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 0x7fec3c10ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:40:47.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.605+0000 7fec3aa00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c10fa40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00b3b0 tx=0x7fec2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:47.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.605+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2c00e030 con 0x7fec3c0fcc30 2024-04-16T11:40:47.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.605+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec2c00bbf0 con 0x7fec3c0fcc30 2024-04-16T11:40:47.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.605+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c111080 con 0x7fec3c0fcc30 2024-04-16T11:40:47.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.605+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c1115c0 con 0x7fec3c0fcc30 2024-04-16T11:40:47.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.605+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2c01c400 con 0x7fec3c0fcc30 2024-04-16T11:40:47.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.606+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c101870 con 0x7fec3c0fcc30 2024-04-16T11:40:47.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.607+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fec2c00bd60 con 0x7fec3c0fcc30 2024-04-16T11:40:47.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.608+0000 7fec32a00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec200692b0 0x7fec2006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:40:47.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.608+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fec2c088e90 con 0x7fec3c0fcc30 2024-04-16T11:40:47.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.609+0000 7fec3a000700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec200692b0 0x7fec2006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:40:47.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.609+0000 7fec3a000700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec200692b0 0x7fec2006b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec28006fd0 tx=0x7fec28008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:40:47.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.614+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fec2c058bb0 con 0x7fec3c0fcc30 2024-04-16T11:40:47.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.952+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fec3c05f5f0 con 0x7fec3c0fcc30 2024-04-16T11:40:47.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.953+0000 7fec32a00700 1 -- 172.21.15.44:0/2814141333 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fec3c05f5f0 con 0x7fec3c0fcc30 2024-04-16T11:40:47.951 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:40:47.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec200692b0 msgr2=0x7fec2006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec200692b0 0x7fec2006b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec28006fd0 tx=0x7fec28008040 comp rx=0 tx=0).stop 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 msgr2=0x7fec3c10fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c10fa40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00b3b0 tx=0x7fec2c009e30 comp rx=0 tx=0).stop 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 shutdown_connections 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fec200692b0 0x7fec2006b760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec3c0fcc30 0x7fec3c10fa40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 --2- 172.21.15.44:0/2814141333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec3c0fdee0 0x7fec3c10ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 >> 172.21.15.44:0/2814141333 conn(0x7fec3c0f83d0 msgr2=0x7fec3c0f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 shutdown_connections 2024-04-16T11:40:47.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:40:47.956+0000 7fec3be00700 1 -- 172.21.15.44:0/2814141333 wait complete. 2024-04-16T11:40:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:48 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2814141333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:40:48.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:48 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2814141333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:40:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:49 smithi044 ceph-mon[28731]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:49 smithi044 ceph-mon[28731]: pgmap v259: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:49 smithi045 ceph-mon[36308]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:40:49.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:49 smithi045 ceph-mon[36308]: pgmap v259: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:51.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:51 smithi044 ceph-mon[28731]: pgmap v260: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:51 smithi045 ceph-mon[36308]: pgmap v260: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:53.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:53 smithi044 ceph-mon[28731]: pgmap v261: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:53 smithi045 ceph-mon[36308]: pgmap v261: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:55 smithi044 ceph-mon[28731]: pgmap v262: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:55 smithi045 ceph-mon[36308]: pgmap v262: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:57 smithi044 ceph-mon[28731]: pgmap v263: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:57 smithi045 ceph-mon[36308]: pgmap v263: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:58.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:40:58 smithi044 ceph-mon[28731]: pgmap v264: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:40:58.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:40:58 smithi045 ceph-mon[36308]: pgmap v264: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:01 smithi045 ceph-mon[36308]: pgmap v265: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:01 smithi044 ceph-mon[28731]: pgmap v265: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:03.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:03 smithi045 ceph-mon[36308]: pgmap v266: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:03.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:03 smithi044 ceph-mon[28731]: pgmap v266: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:05.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:05 smithi045 ceph-mon[36308]: pgmap v267: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:05 smithi044 ceph-mon[28731]: pgmap v267: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:07.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:07 smithi045 ceph-mon[36308]: pgmap v268: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:07 smithi044 ceph-mon[28731]: pgmap v268: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:09 smithi045 ceph-mon[36308]: pgmap v269: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:09.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:09 smithi044 ceph-mon[28731]: pgmap v269: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:11 smithi045 ceph-mon[36308]: pgmap v270: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:11 smithi044 ceph-mon[28731]: pgmap v270: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:13.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:13 smithi044 ceph-mon[28731]: pgmap v271: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:13.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:13 smithi045 ceph-mon[36308]: pgmap v271: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:15.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:15 smithi044 ceph-mon[28731]: pgmap v272: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:15.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:15 smithi045 ceph-mon[36308]: pgmap v272: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:16.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:17 smithi044 ceph-mon[28731]: pgmap v273: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:17.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:17 smithi045 ceph-mon[36308]: pgmap v273: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:18.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 -- 172.21.15.44:0/1498304403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fcc20 msgr2=0x7f331c0fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 --2- 172.21.15.44:0/1498304403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fcc20 0x7f331c0fd030 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f330c004940 tx=0x7f330c009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:18.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 -- 172.21.15.44:0/1498304403 shutdown_connections 2024-04-16T11:41:18.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 --2- 172.21.15.44:0/1498304403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f331c0fded0 0x7f331c0fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 --2- 172.21.15.44:0/1498304403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fcc20 0x7f331c0fd030 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 -- 172.21.15.44:0/1498304403 >> 172.21.15.44:0/1498304403 conn(0x7f331c0f83a0 msgr2=0x7f331c0fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:18.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 -- 172.21.15.44:0/1498304403 shutdown_connections 2024-04-16T11:41:18.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.141+0000 7f331b200700 1 -- 172.21.15.44:0/1498304403 wait complete. 2024-04-16T11:41:18.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.142+0000 7f331b200700 1 Processor -- start 2024-04-16T11:41:18.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.142+0000 7f331b200700 1 -- start start 2024-04-16T11:41:18.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f331b200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f331c0fcc20 0x7f331c10fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f331b200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 0x7f331c10ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f331b200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c110620 con 0x7f331c0fded0 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f331b200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c110e10 con 0x7f331c0fcc20 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f3319400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 0x7f331c10ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f3319400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 0x7f331c10ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33320/0 (socket says 172.21.15.44:33320) 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f3319400700 1 -- 172.21.15.44:0/3573071919 learned_addr learned my addr 172.21.15.44:0/3573071919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f3319e00700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f331c0fcc20 0x7f331c10fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f3319400700 1 -- 172.21.15.44:0/3573071919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f331c0fcc20 msgr2=0x7f331c10fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.143+0000 7f3319400700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f331c0fcc20 0x7f331c10fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f3319400700 1 -- 172.21.15.44:0/3573071919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c005040 con 0x7f331c0fded0 2024-04-16T11:41:18.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f3319400700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 0x7f331c10ff90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f330800b790 tx=0x7f3308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:18.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f330800f770 con 0x7f331c0fded0 2024-04-16T11:41:18.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f330800fda0 con 0x7f331c0fded0 2024-04-16T11:41:18.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c1110f0 con 0x7f331c0fded0 2024-04-16T11:41:18.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f330800d310 con 0x7f331c0fded0 2024-04-16T11:41:18.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.144+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c1116b0 con 0x7f331c0fded0 2024-04-16T11:41:18.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.145+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331c101860 con 0x7f331c0fded0 2024-04-16T11:41:18.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.146+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f3308010420 con 0x7f331c0fded0 2024-04-16T11:41:18.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.147+0000 7f3312000700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f33000692a0 0x7f330006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.147+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3308087260 con 0x7f331c0fded0 2024-04-16T11:41:18.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.148+0000 7f3319e00700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f33000692a0 0x7f330006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.148+0000 7f3319e00700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f33000692a0 0x7f330006b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f330c004f90 tx=0x7f330c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:18.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.153+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3308056f80 con 0x7f331c0fded0 2024-04-16T11:41:18.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.390+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f331c105900 con 0x7f33000692a0 2024-04-16T11:41:18.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.393+0000 7f3312000700 1 -- 172.21.15.44:0/3573071919 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f331c105900 con 0x7f33000692a0 2024-04-16T11:41:18.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.395+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f33000692a0 msgr2=0x7f330006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.395+0000 7f331b200700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f33000692a0 0x7f330006b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f330c004f90 tx=0x7f330c009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.395+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 msgr2=0x7f331c10ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.395+0000 7f331b200700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 0x7f331c10ff90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f330800b790 tx=0x7f3308009250 comp rx=0 tx=0).stop 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 shutdown_connections 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f33000692a0 0x7f330006b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f331c0fcc20 0x7f331c10fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 --2- 172.21.15.44:0/3573071919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c0fded0 0x7f331c10ff90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 >> 172.21.15.44:0/3573071919 conn(0x7f331c0f83a0 msgr2=0x7f331c0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:18.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 shutdown_connections 2024-04-16T11:41:18.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.396+0000 7f331b200700 1 -- 172.21.15.44:0/3573071919 wait complete. 2024-04-16T11:41:18.411 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.559+0000 7f8586400700 1 -- 172.21.15.44:0/4172817910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 msgr2=0x7f85800703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.559+0000 7f8586400700 1 --2- 172.21.15.44:0/4172817910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f85800703b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8570004940 tx=0x7f8570009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.559+0000 7f8586400700 1 -- 172.21.15.44:0/4172817910 shutdown_connections 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.559+0000 7f8586400700 1 --2- 172.21.15.44:0/4172817910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85800708f0 0x7f8580070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.559+0000 7f8586400700 1 --2- 172.21.15.44:0/4172817910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f85800703b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.559+0000 7f8586400700 1 -- 172.21.15.44:0/4172817910 >> 172.21.15.44:0/4172817910 conn(0x7f85800f8580 msgr2=0x7f85800fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.560+0000 7f8586400700 1 -- 172.21.15.44:0/4172817910 shutdown_connections 2024-04-16T11:41:18.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.560+0000 7f8586400700 1 -- 172.21.15.44:0/4172817910 wait complete. 2024-04-16T11:41:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.560+0000 7f8586400700 1 Processor -- start 2024-04-16T11:41:18.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8586400700 1 -- start start 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8586400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85800708f0 0x7f858010fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8586400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f8580110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8586400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8580110700 con 0x7f8580071ea0 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8586400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8580110870 con 0x7f85800708f0 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8576a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f8580110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8576a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f8580110100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33348/0 (socket says 172.21.15.44:33348) 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.561+0000 7f8576a00700 1 -- 172.21.15.44:0/2795835995 learned_addr learned my addr 172.21.15.44:0/2795835995 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:18.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.562+0000 7f857f400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85800708f0 0x7f858010fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.562+0000 7f8576a00700 1 -- 172.21.15.44:0/2795835995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85800708f0 msgr2=0x7f858010fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.562+0000 7f8576a00700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85800708f0 0x7f858010fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.562+0000 7f8576a00700 1 -- 172.21.15.44:0/2795835995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8570005040 con 0x7f8580071ea0 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.562+0000 7f8576a00700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f8580110100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8568002a50 tx=0x7f85680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.562+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8568007cc0 con 0x7f8580071ea0 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.563+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f856800f040 con 0x7f8580071ea0 2024-04-16T11:41:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.563+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85801152d0 con 0x7f8580071ea0 2024-04-16T11:41:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.563+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8568009640 con 0x7f8580071ea0 2024-04-16T11:41:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.563+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8580115890 con 0x7f8580071ea0 2024-04-16T11:41:18.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.564+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858004c710 con 0x7f8580071ea0 2024-04-16T11:41:18.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.565+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f85680097a0 con 0x7f8580071ea0 2024-04-16T11:41:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.566+0000 7f8577e00700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f85640692a0 0x7f856406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.566+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8568086f60 con 0x7f8580071ea0 2024-04-16T11:41:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.566+0000 7f857f400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f85640692a0 0x7f856406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.567+0000 7f857f400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f85640692a0 0x7f856406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f857000b970 tx=0x7f8570009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:18.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.570+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8568056c80 con 0x7f8580071ea0 2024-04-16T11:41:18.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.811+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8580103ea0 con 0x7f85640692a0 2024-04-16T11:41:18.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.813+0000 7f8577e00700 1 -- 172.21.15.44:0/2795835995 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f8580103ea0 con 0x7f85640692a0 2024-04-16T11:41:18.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.815+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f85640692a0 msgr2=0x7f856406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.815+0000 7f8586400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f85640692a0 0x7f856406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f857000b970 tx=0x7f8570009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.815+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 msgr2=0x7f8580110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.815+0000 7f8586400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f8580110100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8568002a50 tx=0x7f85680046b0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 shutdown_connections 2024-04-16T11:41:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f85640692a0 0x7f856406b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85800708f0 0x7f858010fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 --2- 172.21.15.44:0/2795835995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8580071ea0 0x7f8580110100 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 >> 172.21.15.44:0/2795835995 conn(0x7f85800f8580 msgr2=0x7f85800f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 shutdown_connections 2024-04-16T11:41:18.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.816+0000 7f8586400700 1 -- 172.21.15.44:0/2795835995 wait complete. 2024-04-16T11:41:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 -- 172.21.15.44:0/1089889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 msgr2=0x7f49ec0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 --2- 172.21.15.44:0/1089889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec0fd320 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f49e0004940 tx=0x7f49e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:18.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 -- 172.21.15.44:0/1089889111 shutdown_connections 2024-04-16T11:41:18.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 --2- 172.21.15.44:0/1089889111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49ec0fe0e0 0x7f49ec0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 --2- 172.21.15.44:0/1089889111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec0fd320 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 -- 172.21.15.44:0/1089889111 >> 172.21.15.44:0/1089889111 conn(0x7f49ec0f8530 msgr2=0x7f49ec0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:18.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.975+0000 7f49f3a00700 1 -- 172.21.15.44:0/1089889111 shutdown_connections 2024-04-16T11:41:18.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.976+0000 7f49f3a00700 1 -- 172.21.15.44:0/1089889111 wait complete. 2024-04-16T11:41:18.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.977+0000 7f49f3a00700 1 Processor -- start 2024-04-16T11:41:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.977+0000 7f49f3a00700 1 -- start start 2024-04-16T11:41:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.977+0000 7f49f3a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.977+0000 7f49f3a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49ec0fe0e0 0x7f49ec110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.977+0000 7f49f3a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec110870 con 0x7f49ec0fcef0 2024-04-16T11:41:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.977+0000 7f49f3a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec110fb0 con 0x7f49ec0fe0e0 2024-04-16T11:41:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec10fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33380/0 (socket says 172.21.15.44:33380) 2024-04-16T11:41:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 -- 172.21.15.44:0/2878447143 learned_addr learned my addr 172.21.15.44:0/2878447143 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49ebe00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49ec0fe0e0 0x7f49ec110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 -- 172.21.15.44:0/2878447143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49ec0fe0e0 msgr2=0x7f49ec110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49ec0fe0e0 0x7f49ec110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 -- 172.21.15.44:0/2878447143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e0005040 con 0x7f49ec0fcef0 2024-04-16T11:41:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.978+0000 7f49f1000700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec10fd30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49e000b970 tx=0x7f49e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.979+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49e000e040 con 0x7f49ec0fcef0 2024-04-16T11:41:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.979+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49e000bbf0 con 0x7f49ec0fcef0 2024-04-16T11:41:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.979+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec111230 con 0x7f49ec0fcef0 2024-04-16T11:41:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.979+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49e001c630 con 0x7f49ec0fcef0 2024-04-16T11:41:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.979+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec1117f0 con 0x7f49ec0fcef0 2024-04-16T11:41:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.980+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec101a70 con 0x7f49ec0fcef0 2024-04-16T11:41:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.982+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f49e000bd60 con 0x7f49ec0fcef0 2024-04-16T11:41:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.983+0000 7f49e8c00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49d40692b0 0x7f49d406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.983+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f49e0088560 con 0x7f49ec0fcef0 2024-04-16T11:41:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.983+0000 7f49ebe00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49d40692b0 0x7f49d406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:18.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.984+0000 7f49ebe00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49d40692b0 0x7f49d406b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49d8004d60 tx=0x7f49d800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:18.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:18.988+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f49e00583d0 con 0x7f49ec0fcef0 2024-04-16T11:41:19.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.218+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f49ec111a10 con 0x7f49d40692b0 2024-04-16T11:41:19.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.248+0000 7f49e8c00700 1 -- 172.21.15.44:0/2878447143 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f49ec111a10 con 0x7f49d40692b0 2024-04-16T11:41:19.246 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:41:19.246 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (7m) 74s ago 8m 13.5M - 0.25.0 c8568f914cd2 8136acabd663 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (9m) 74s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (7m) 80s ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (6m) 74s ago 8m 75.2M - 9.4.7 954c08fa6188 afde3dcbfd10 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (99s) 74s ago 99s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (107s) 74s ago 107s 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (96s) 80s ago 95s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (104s) 80s ago 103s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:9283 running (10m) 74s ago 10m 483M - 17.2.7-847-g45317f95 b333581ccdf5 53edec877549 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (7m) 80s ago 7m 432M - 17.2.7-847-g45317f95 b333581ccdf5 f6be9ec13da1 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (10m) 74s ago 10m 43.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (7m) 80s ago 7m 34.6M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (9m) 74s ago 9m 7746k - 1.5.0 0da6a335fe13 dbe9480d7dd0 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (7m) 80s ago 7m 7486k - 1.5.0 0da6a335fe13 e37d89867e75 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (6m) 74s ago 6m 44.5M 4290M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (5m) 74s ago 5m 46.9M 4290M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:41:19.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (5m) 74s ago 5m 43.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:41:19.248 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (4m) 80s ago 4m 42.9M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:41:19.248 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (4m) 80s ago 4m 40.6M 5314M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:41:19.248 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (3m) 80s ago 3m 42.6M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:41:19.248 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (6m) 74s ago 8m 44.5M - 2.43.0 a07b618ecd1d 99d239fdb386 2024-04-16T11:41:19.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49d40692b0 msgr2=0x7f49d406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49d40692b0 0x7f49d406b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49d8004d60 tx=0x7f49d800b040 comp rx=0 tx=0).stop 2024-04-16T11:41:19.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 msgr2=0x7f49ec10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec10fd30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49e000b970 tx=0x7f49e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 shutdown_connections 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f49d40692b0 0x7f49d406b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49ec0fcef0 0x7f49ec10fd30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 --2- 172.21.15.44:0/2878447143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49ec0fe0e0 0x7f49ec110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 >> 172.21.15.44:0/2878447143 conn(0x7f49ec0f8530 msgr2=0x7f49ec0f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.251+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 shutdown_connections 2024-04-16T11:41:19.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.252+0000 7f49f3a00700 1 -- 172.21.15.44:0/2878447143 wait complete. 2024-04-16T11:41:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:19 smithi045 ceph-mon[36308]: pgmap v274: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:41:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:41:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:19 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:41:19.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.409+0000 7f7b14400700 1 -- 172.21.15.44:0/1374618071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 msgr2=0x7f7b0c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.409+0000 7f7b14400700 1 --2- 172.21.15.44:0/1374618071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c0fe070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7af8004940 tx=0x7f7af8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:19 smithi044 ceph-mon[28731]: pgmap v274: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:41:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:41:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:19 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:41:19.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.410+0000 7f7b14400700 1 -- 172.21.15.44:0/1374618071 shutdown_connections 2024-04-16T11:41:19.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.410+0000 7f7b14400700 1 --2- 172.21.15.44:0/1374618071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c0fe070 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.410+0000 7f7b14400700 1 --2- 172.21.15.44:0/1374618071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c0fcdc0 0x7f7b0c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.410+0000 7f7b14400700 1 -- 172.21.15.44:0/1374618071 >> 172.21.15.44:0/1374618071 conn(0x7f7b0c0f8540 msgr2=0x7f7b0c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:19.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.410+0000 7f7b14400700 1 -- 172.21.15.44:0/1374618071 shutdown_connections 2024-04-16T11:41:19.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.410+0000 7f7b14400700 1 -- 172.21.15.44:0/1374618071 wait complete. 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.411+0000 7f7b14400700 1 Processor -- start 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.411+0000 7f7b14400700 1 -- start start 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.411+0000 7f7b14400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c0fcdc0 0x7f7b0c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.411+0000 7f7b14400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.411+0000 7f7b14400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0708c0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.411+0000 7f7b14400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c070a30 con 0x7f7b0c0fcdc0 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33400/0 (socket says 172.21.15.44:33400) 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11000700 1 -- 172.21.15.44:0/260729260 learned_addr learned my addr 172.21.15.44:0/260729260 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:19.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11a00700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c0fcdc0 0x7f7b0c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11000700 1 -- 172.21.15.44:0/260729260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c0fcdc0 msgr2=0x7f7b0c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11000700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c0fcdc0 0x7f7b0c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.412+0000 7f7b11000700 1 -- 172.21.15.44:0/260729260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8005040 con 0x7f7b0c0fd710 2024-04-16T11:41:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.413+0000 7f7b11000700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c070310 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7af800b970 tx=0x7f7af8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.413+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7af800e040 con 0x7f7b0c0fd710 2024-04-16T11:41:19.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.413+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c070cb0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.413+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7af800bbf0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.413+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7af801c3f0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.413+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c115ae0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.414+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c05f5f0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.415+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7af800bd60 con 0x7f7b0c0fd710 2024-04-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.416+0000 7f7b09600700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7af40692b0 0x7f7af406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.416+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7af8088470 con 0x7f7b0c0fd710 2024-04-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.417+0000 7f7b11a00700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7af40692b0 0x7f7af406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.417+0000 7f7b11a00700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7af40692b0 0x7f7af406b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c071bd0 tx=0x7f7b0000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:19.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.422+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7af8054830 con 0x7f7b0c0fd710 2024-04-16T11:41:19.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.725+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b0c04bed0 con 0x7f7b0c0fd710 2024-04-16T11:41:19.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.726+0000 7f7b09600700 1 -- 172.21.15.44:0/260729260 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f7af8057e50 con 0x7f7b0c0fd710 2024-04-16T11:41:19.724 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:41:19.724 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:41:19.724 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:41:19.725 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7af40692b0 msgr2=0x7f7af406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7af40692b0 0x7f7af406b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c071bd0 tx=0x7f7b0000b040 comp rx=0 tx=0).stop 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 msgr2=0x7f7b0c070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c070310 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7af800b970 tx=0x7f7af8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 shutdown_connections 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7af40692b0 0x7f7af406b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c0fcdc0 0x7f7b0c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 --2- 172.21.15.44:0/260729260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0fd710 0x7f7b0c070310 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 >> 172.21.15.44:0/260729260 conn(0x7f7b0c0f8540 msgr2=0x7f7b0c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 shutdown_connections 2024-04-16T11:41:19.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.729+0000 7f7b14400700 1 -- 172.21.15.44:0/260729260 wait complete. 2024-04-16T11:41:19.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.886+0000 7fd665800700 1 -- 172.21.15.44:0/3373879956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600fff70 msgr2=0x7fd660102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.886+0000 7fd665800700 1 --2- 172.21.15.44:0/3373879956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600fff70 0x7fd660102390 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd64c004940 tx=0x7fd64c009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.886+0000 7fd665800700 1 -- 172.21.15.44:0/3373879956 shutdown_connections 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.886+0000 7fd665800700 1 --2- 172.21.15.44:0/3373879956 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600fff70 0x7fd660102390 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.886+0000 7fd665800700 1 --2- 172.21.15.44:0/3373879956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6600662f0 0x7fd6600ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.886+0000 7fd665800700 1 -- 172.21.15.44:0/3373879956 >> 172.21.15.44:0/3373879956 conn(0x7fd6600f74c0 msgr2=0x7fd6600f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.887+0000 7fd665800700 1 -- 172.21.15.44:0/3373879956 shutdown_connections 2024-04-16T11:41:19.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.887+0000 7fd665800700 1 -- 172.21.15.44:0/3373879956 wait complete. 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.887+0000 7fd665800700 1 Processor -- start 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd665800700 1 -- start start 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd665800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 0x7fd66010fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd665800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6600fff70 0x7fd660110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd665800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660110850 con 0x7fd6600662f0 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd665800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660110f90 con 0x7fd6600fff70 2024-04-16T11:41:19.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd65ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 0x7fd66010fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd65ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 0x7fd66010fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33422/0 (socket says 172.21.15.44:33422) 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.888+0000 7fd65ea00700 1 -- 172.21.15.44:0/4056781262 learned_addr learned my addr 172.21.15.44:0/4056781262 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd65e000700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6600fff70 0x7fd660110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd65ea00700 1 -- 172.21.15.44:0/4056781262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6600fff70 msgr2=0x7fd660110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd65ea00700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6600fff70 0x7fd660110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd65ea00700 1 -- 172.21.15.44:0/4056781262 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c005040 con 0x7fd6600662f0 2024-04-16T11:41:19.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd65ea00700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 0x7fd66010fd10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd650002a50 tx=0x7fd650004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd650007cc0 con 0x7fd6600662f0 2024-04-16T11:41:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd65000f040 con 0x7fd6600662f0 2024-04-16T11:41:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd65000daf0 con 0x7fd6600662f0 2024-04-16T11:41:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.889+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd660111270 con 0x7fd6600662f0 2024-04-16T11:41:19.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.890+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd660111830 con 0x7fd6600662f0 2024-04-16T11:41:19.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.891+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66005f5f0 con 0x7fd6600662f0 2024-04-16T11:41:19.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.892+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd650009810 con 0x7fd6600662f0 2024-04-16T11:41:19.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.893+0000 7fd656a00700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd6440692a0 0x7fd64406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:19.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.893+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd650086f10 con 0x7fd6600662f0 2024-04-16T11:41:19.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.893+0000 7fd65e000700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd6440692a0 0x7fd64406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.894+0000 7fd65e000700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd6440692a0 0x7fd64406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00b3b0 tx=0x7fd64c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:19.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:19.897+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd650056c30 con 0x7fd6600662f0 2024-04-16T11:41:20.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.163+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd660111c00 con 0x7fd6600662f0 2024-04-16T11:41:20.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.164+0000 7fd656a00700 1 -- 172.21.15.44:0/4056781262 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fd6500567c0 con 0x7fd6600662f0 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:41:20.163 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:41:20.164 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:20.165 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.166+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd6440692a0 msgr2=0x7fd64406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.166+0000 7fd665800700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd6440692a0 0x7fd64406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00b3b0 tx=0x7fd64c009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.166+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 msgr2=0x7fd66010fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.166+0000 7fd665800700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 0x7fd66010fd10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd650002a50 tx=0x7fd650004600 comp rx=0 tx=0).stop 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 shutdown_connections 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fd6440692a0 0x7fd64406b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6600662f0 0x7fd66010fd10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 --2- 172.21.15.44:0/4056781262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6600fff70 0x7fd660110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 >> 172.21.15.44:0/4056781262 conn(0x7fd6600f74c0 msgr2=0x7fd6600fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:20.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 shutdown_connections 2024-04-16T11:41:20.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.167+0000 7fd665800700 1 -- 172.21.15.44:0/4056781262 wait complete. 2024-04-16T11:41:20.167 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 -- 172.21.15.44:0/3037250550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1041046d0 msgr2=0x7fb104104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 --2- 172.21.15.44:0/3037250550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1041046d0 0x7fb104104b40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0004880 tx=0x7fb0f0009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 -- 172.21.15.44:0/3037250550 shutdown_connections 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 --2- 172.21.15.44:0/3037250550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1041046d0 0x7fb104104b40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 --2- 172.21.15.44:0/3037250550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb104103420 0x7fb104103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 -- 172.21.15.44:0/3037250550 >> 172.21.15.44:0/3037250550 conn(0x7fb104072640 msgr2=0x7fb104074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:20.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 -- 172.21.15.44:0/3037250550 shutdown_connections 2024-04-16T11:41:20.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.321+0000 7fb109a00700 1 -- 172.21.15.44:0/3037250550 wait complete. 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb109a00700 1 Processor -- start 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb109a00700 1 -- start start 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb109a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104103420 0x7fb104113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb109a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 0x7fb1041144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb109a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb104114aa0 con 0x7fb1041046d0 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb109a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb104114c10 con 0x7fb104103420 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb102000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 0x7fb1041144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb102000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 0x7fb1041144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33440/0 (socket says 172.21.15.44:33440) 2024-04-16T11:41:20.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.322+0000 7fb102000700 1 -- 172.21.15.44:0/3954198948 learned_addr learned my addr 172.21.15.44:0/3954198948 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:20.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb102a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104103420 0x7fb104113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:20.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb102000700 1 -- 172.21.15.44:0/3954198948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104103420 msgr2=0x7fb104113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb102000700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104103420 0x7fb104113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb102000700 1 -- 172.21.15.44:0/3954198948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f0005040 con 0x7fb1041046d0 2024-04-16T11:41:20.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb102000700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 0x7fb1041144a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0008fb0 tx=0x7fb0f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:20.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f000e030 con 0x7fb1041046d0 2024-04-16T11:41:20.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0f0009430 con 0x7fb1041046d0 2024-04-16T11:41:20.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f00123c0 con 0x7fb1041046d0 2024-04-16T11:41:20.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.323+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb104119610 con 0x7fb1041046d0 2024-04-16T11:41:20.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.324+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb104119bd0 con 0x7fb1041046d0 2024-04-16T11:41:20.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.325+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10404c710 con 0x7fb1041046d0 2024-04-16T11:41:20.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.326+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb0f0013b70 con 0x7fb1041046d0 2024-04-16T11:41:20.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.326+0000 7fb0faa00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb0e80692a0 0x7fb0e806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:20.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.327+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb0f0088bb0 con 0x7fb1041046d0 2024-04-16T11:41:20.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.327+0000 7fb102a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb0e80692a0 0x7fb0e806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:20.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.328+0000 7fb102a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb0e80692a0 0x7fb0e806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40046c0 tx=0x7fb0f400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:20.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.332+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb0f00588d0 con 0x7fb1041046d0 2024-04-16T11:41:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:20 smithi044 ceph-mon[28731]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:20 smithi044 ceph-mon[28731]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:20 smithi044 ceph-mon[28731]: Upgrade: Target is version 19.0.0-2884-g218a6914 (unknown) 2024-04-16T11:41:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:20 smithi044 ceph-mon[28731]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa'] 2024-04-16T11:41:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:20 smithi044 ceph-mon[28731]: Upgrade: Need to upgrade myself (mgr.smithi044.sczobv) 2024-04-16T11:41:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:20 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/260729260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:41:20.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.563+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb104108a30 con 0x7fb0e80692a0 2024-04-16T11:41:20.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.566+0000 7fb0faa00700 1 -- 172.21.15.44:0/3954198948 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fb104108a30 con 0x7fb0e80692a0 2024-04-16T11:41:20.564 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:41:20.564 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:41:20.564 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:41:20.564 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:41:20.564 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:41:20.565 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "0/21 daemons upgraded", 2024-04-16T11:41:20.565 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa image on host smithi045", 2024-04-16T11:41:20.565 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:41:20.565 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.568+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb0e80692a0 msgr2=0x7fb0e806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.568+0000 7fb109a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb0e80692a0 0x7fb0e806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40046c0 tx=0x7fb0f400b040 comp rx=0 tx=0).stop 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.568+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 msgr2=0x7fb1041144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.568+0000 7fb109a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 0x7fb1041144a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0008fb0 tx=0x7fb0f0009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 shutdown_connections 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7fb0e80692a0 0x7fb0e806b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb104103420 0x7fb104113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 --2- 172.21.15.44:0/3954198948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb1041046d0 0x7fb1041144a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 >> 172.21.15.44:0/3954198948 conn(0x7fb104072640 msgr2=0x7fb104073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:20.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 shutdown_connections 2024-04-16T11:41:20.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.569+0000 7fb109a00700 1 -- 172.21.15.44:0/3954198948 wait complete. 2024-04-16T11:41:20.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:20 smithi045 ceph-mon[36308]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:20.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:20 smithi045 ceph-mon[36308]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:20.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:20 smithi045 ceph-mon[36308]: Upgrade: Target is version 19.0.0-2884-g218a6914 (unknown) 2024-04-16T11:41:20.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:20 smithi045 ceph-mon[36308]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa'] 2024-04-16T11:41:20.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:20 smithi045 ceph-mon[36308]: Upgrade: Need to upgrade myself (mgr.smithi044.sczobv) 2024-04-16T11:41:20.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:20 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/260729260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.717+0000 7f7b48c00700 1 -- 172.21.15.44:0/2238540278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 msgr2=0x7f7b440fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.717+0000 7f7b48c00700 1 --2- 172.21.15.44:0/2238540278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b440fd040 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7b34004940 tx=0x7f7b34009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.718+0000 7f7b48c00700 1 -- 172.21.15.44:0/2238540278 shutdown_connections 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.718+0000 7f7b48c00700 1 --2- 172.21.15.44:0/2238540278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 0x7f7b440fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.718+0000 7f7b48c00700 1 --2- 172.21.15.44:0/2238540278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b440fd040 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.718+0000 7f7b48c00700 1 -- 172.21.15.44:0/2238540278 >> 172.21.15.44:0/2238540278 conn(0x7f7b440f83d0 msgr2=0x7f7b440fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.718+0000 7f7b48c00700 1 -- 172.21.15.44:0/2238540278 shutdown_connections 2024-04-16T11:41:20.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.718+0000 7f7b48c00700 1 -- 172.21.15.44:0/2238540278 wait complete. 2024-04-16T11:41:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b48c00700 1 Processor -- start 2024-04-16T11:41:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b48c00700 1 -- start start 2024-04-16T11:41:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b48c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b4410fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b48c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 0x7f7b4410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b48c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44110630 con 0x7f7b440fcc30 2024-04-16T11:41:20.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b48c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44110e20 con 0x7f7b440fdee0 2024-04-16T11:41:20.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b43400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b4410fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:20.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b43400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b4410fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33450/0 (socket says 172.21.15.44:33450) 2024-04-16T11:41:20.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.719+0000 7f7b43400700 1 -- 172.21.15.44:0/3635884616 learned_addr learned my addr 172.21.15.44:0/3635884616 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b43400700 1 -- 172.21.15.44:0/3635884616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 msgr2=0x7f7b4410ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b42a00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 0x7f7b4410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b43400700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 0x7f7b4410ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b43400700 1 -- 172.21.15.44:0/3635884616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b34005040 con 0x7f7b440fcc30 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b42a00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 0x7f7b4410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b43400700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b4410fa60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400b3b0 tx=0x7f7b34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3400e040 con 0x7f7b440fcc30 2024-04-16T11:41:20.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.720+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b34004290 con 0x7f7b440fcc30 2024-04-16T11:41:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.721+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b441110a0 con 0x7f7b440fcc30 2024-04-16T11:41:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.721+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3401c630 con 0x7f7b440fcc30 2024-04-16T11:41:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.721+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b44111660 con 0x7f7b440fcc30 2024-04-16T11:41:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.722+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44101870 con 0x7f7b440fcc30 2024-04-16T11:41:20.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.723+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f7b3400bbf0 con 0x7f7b440fcc30 2024-04-16T11:41:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.723+0000 7f7b3b400700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7b280692b0 0x7f7b2806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:20.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.724+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7b34088610 con 0x7f7b440fcc30 2024-04-16T11:41:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.724+0000 7f7b42a00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7b280692b0 0x7f7b2806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:20.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.725+0000 7f7b42a00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7b280692b0 0x7f7b2806b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:20.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:20.729+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7b340583d0 con 0x7f7b440fcc30 2024-04-16T11:41:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.023+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7b4404bed0 con 0x7f7b440fcc30 2024-04-16T11:41:21.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.024+0000 7f7b3b400700 1 -- 172.21.15.44:0/3635884616 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f7b34057f60 con 0x7f7b440fcc30 2024-04-16T11:41:21.022 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:41:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.026+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7b280692b0 msgr2=0x7f7b2806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:21.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.026+0000 7f7b48c00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7b280692b0 0x7f7b2806b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30008040 comp rx=0 tx=0).stop 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.026+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 msgr2=0x7f7b4410fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.026+0000 7f7b48c00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b4410fa60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400b3b0 tx=0x7f7b34009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.026+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 shutdown_connections 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.027+0000 7f7b48c00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f7b280692b0 0x7f7b2806b760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.027+0000 7f7b48c00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b440fcc30 0x7f7b4410fa60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.027+0000 7f7b48c00700 1 --2- 172.21.15.44:0/3635884616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b440fdee0 0x7f7b4410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.027+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 >> 172.21.15.44:0/3635884616 conn(0x7f7b440f83d0 msgr2=0x7f7b440f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:21.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.027+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 shutdown_connections 2024-04-16T11:41:21.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:21.027+0000 7f7b48c00700 1 -- 172.21.15.44:0/3635884616 wait complete. 2024-04-16T11:41:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:21 smithi044 ceph-mon[28731]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:21 smithi044 ceph-mon[28731]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa on smithi045 2024-04-16T11:41:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:21 smithi044 ceph-mon[28731]: pgmap v275: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:21 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/4056781262' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:41:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:21 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3635884616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:41:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:21 smithi045 ceph-mon[36308]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:21 smithi045 ceph-mon[36308]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa on smithi045 2024-04-16T11:41:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:21 smithi045 ceph-mon[36308]: pgmap v275: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:21 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/4056781262' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:41:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:21 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3635884616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:41:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:22 smithi044 ceph-mon[28731]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:22.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:22 smithi045 ceph-mon[36308]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:23 smithi044 ceph-mon[28731]: pgmap v276: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:23 smithi045 ceph-mon[36308]: pgmap v276: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:25 smithi044 ceph-mon[28731]: pgmap v277: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:25.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:25 smithi045 ceph-mon[36308]: pgmap v277: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:27 smithi044 ceph-mon[28731]: pgmap v278: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:27.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:27 smithi045 ceph-mon[36308]: pgmap v278: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:29.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:29 smithi044 ceph-mon[28731]: pgmap v279: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:29.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:29 smithi045 ceph-mon[36308]: pgmap v279: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:31.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:31 smithi045 ceph-mon[36308]: pgmap v280: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:31.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:31 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:31 smithi044 ceph-mon[28731]: pgmap v280: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:31.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:31 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:33.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:33 smithi044 ceph-mon[28731]: pgmap v281: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:33.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:33 smithi045 ceph-mon[36308]: pgmap v281: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:35 smithi044 ceph-mon[28731]: pgmap v282: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:35.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:35 smithi045 ceph-mon[36308]: pgmap v282: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:36.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:36 smithi045 ceph-mon[36308]: pgmap v283: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:36.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:36 smithi044 ceph-mon[28731]: pgmap v283: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:39.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:39 smithi045 ceph-mon[36308]: pgmap v284: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:39.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:39 smithi044 ceph-mon[28731]: pgmap v284: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:41.320 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:41 smithi045 ceph-mon[36308]: pgmap v285: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:41.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:41 smithi044 ceph-mon[28731]: pgmap v285: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:43.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:43 smithi045 ceph-mon[36308]: pgmap v286: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:43 smithi044 ceph-mon[28731]: pgmap v286: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:45.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:45 smithi045 ceph-mon[36308]: pgmap v287: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:45 smithi044 ceph-mon[28731]: pgmap v287: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:46.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:46 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:46.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:46 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:41:47.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:47 smithi044 ceph-mon[28731]: pgmap v288: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:47.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:47 smithi045 ceph-mon[36308]: pgmap v288: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:49.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:49 smithi045 ceph-mon[36308]: pgmap v289: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:49 smithi044 ceph-mon[28731]: pgmap v289: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 -- 172.21.15.44:0/1912623880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 msgr2=0x7ff788102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 --2- 172.21.15.44:0/1912623880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff788102390 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff774004940 tx=0x7ff774009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 -- 172.21.15.44:0/1912623880 shutdown_connections 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 --2- 172.21.15.44:0/1912623880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff788102390 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 --2- 172.21.15.44:0/1912623880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7880662f0 0x7ff7880ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 -- 172.21.15.44:0/1912623880 >> 172.21.15.44:0/1912623880 conn(0x7ff7880f74c0 msgr2=0x7ff7880f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 -- 172.21.15.44:0/1912623880 shutdown_connections 2024-04-16T11:41:51.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.200+0000 7ff78f800700 1 -- 172.21.15.44:0/1912623880 wait complete. 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.201+0000 7ff78f800700 1 Processor -- start 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.202+0000 7ff78f800700 1 -- start start 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.202+0000 7ff78f800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7880662f0 0x7ff78810fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.202+0000 7ff78f800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff7881101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.202+0000 7ff78f800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7881107a0 con 0x7ff7880fff70 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.202+0000 7ff78f800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788110ff0 con 0x7ff7880662f0 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff7881101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff7881101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35060/0 (socket says 172.21.15.44:35060) 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 -- 172.21.15.44:0/1661413674 learned_addr learned my addr 172.21.15.44:0/1661413674 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:51.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff78ce00700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7880662f0 0x7ff78810fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:51.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 -- 172.21.15.44:0/1661413674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7880662f0 msgr2=0x7ff78810fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7880662f0 0x7ff78810fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 -- 172.21.15.44:0/1661413674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff774005040 con 0x7ff7880fff70 2024-04-16T11:41:51.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.203+0000 7ff787e00700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff7881101a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff77400b3b0 tx=0x7ff774009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:51.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.204+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77400e040 con 0x7ff7880fff70 2024-04-16T11:41:51.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.204+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff774004290 con 0x7ff7880fff70 2024-04-16T11:41:51.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.204+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77401c630 con 0x7ff7880fff70 2024-04-16T11:41:51.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.204+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788111270 con 0x7ff7880fff70 2024-04-16T11:41:51.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.204+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7881117b0 con 0x7ff7880fff70 2024-04-16T11:41:51.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.205+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7880f8ff0 con 0x7ff7880fff70 2024-04-16T11:41:51.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.206+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff77400bbf0 con 0x7ff7880fff70 2024-04-16T11:41:51.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.207+0000 7ff784c00700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7700692b0 0x7ff77006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:51.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.207+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff774088610 con 0x7ff7880fff70 2024-04-16T11:41:51.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.207+0000 7ff78ce00700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7700692b0 0x7ff77006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:51.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.208+0000 7ff78ce00700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7700692b0 0x7ff77006b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff788066030 tx=0x7ff77c009d70 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:51.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.212+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff7740583b0 con 0x7ff7880fff70 2024-04-16T11:41:51.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.445+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff788111a20 con 0x7ff7700692b0 2024-04-16T11:41:51.444 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:51 smithi044 ceph-mon[28731]: pgmap v290: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:51.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.448+0000 7ff784c00700 1 -- 172.21.15.44:0/1661413674 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7ff788111a20 con 0x7ff7700692b0 2024-04-16T11:41:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.450+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7700692b0 msgr2=0x7ff77006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.450+0000 7ff78f800700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7700692b0 0x7ff77006b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff788066030 tx=0x7ff77c009d70 comp rx=0 tx=0).stop 2024-04-16T11:41:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.450+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 msgr2=0x7ff7881101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.450+0000 7ff78f800700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff7881101a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff77400b3b0 tx=0x7ff774009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 shutdown_connections 2024-04-16T11:41:51.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff7700692b0 0x7ff77006b760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7880662f0 0x7ff78810fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 --2- 172.21.15.44:0/1661413674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7880fff70 0x7ff7881101a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 >> 172.21.15.44:0/1661413674 conn(0x7ff7880f74c0 msgr2=0x7ff7880fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 shutdown_connections 2024-04-16T11:41:51.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.451+0000 7ff78f800700 1 -- 172.21.15.44:0/1661413674 wait complete. 2024-04-16T11:41:51.464 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:41:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:51 smithi045 ceph-mon[36308]: pgmap v290: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 -- 172.21.15.44:0/3522593969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e01046d0 msgr2=0x7ff8e0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 --2- 172.21.15.44:0/3522593969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e01046d0 0x7ff8e0104b40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8004940 tx=0x7ff8c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 -- 172.21.15.44:0/3522593969 shutdown_connections 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 --2- 172.21.15.44:0/3522593969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e01046d0 0x7ff8e0104b40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 --2- 172.21.15.44:0/3522593969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e0103420 0x7ff8e0103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 -- 172.21.15.44:0/3522593969 >> 172.21.15.44:0/3522593969 conn(0x7ff8e0072640 msgr2=0x7ff8e0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 -- 172.21.15.44:0/3522593969 shutdown_connections 2024-04-16T11:41:51.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.605+0000 7ff8e7800700 1 -- 172.21.15.44:0/3522593969 wait complete. 2024-04-16T11:41:51.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.606+0000 7ff8e7800700 1 Processor -- start 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.606+0000 7ff8e7800700 1 -- start start 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.606+0000 7ff8e7800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 0x7ff8e0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.606+0000 7ff8e7800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e01046d0 0x7ff8e01144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.606+0000 7ff8e7800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e0114aa0 con 0x7ff8e0103420 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.606+0000 7ff8e7800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e0114c10 con 0x7ff8e01046d0 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 0x7ff8e0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 0x7ff8e0113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33828/0 (socket says 172.21.15.44:33828) 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 -- 172.21.15.44:0/1697793605 learned_addr learned my addr 172.21.15.44:0/1697793605 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:51.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8d7e00700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e01046d0 0x7ff8e01144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 -- 172.21.15.44:0/1697793605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e01046d0 msgr2=0x7ff8e01144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e01046d0 0x7ff8e01144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 -- 172.21.15.44:0/1697793605 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c8005040 con 0x7ff8e0103420 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.607+0000 7ff8e4e00700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 0x7ff8e0113f60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c8c0 tx=0x7ff8d00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.608+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8d000a980 con 0x7ff8e0103420 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.608+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8d000ce60 con 0x7ff8e0103420 2024-04-16T11:41:51.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.608+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e0119670 con 0x7ff8e0103420 2024-04-16T11:41:51.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.608+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8d00115e0 con 0x7ff8e0103420 2024-04-16T11:41:51.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.608+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e0119c30 con 0x7ff8e0103420 2024-04-16T11:41:51.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.609+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e004c710 con 0x7ff8e0103420 2024-04-16T11:41:51.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.610+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff8d0012400 con 0x7ff8e0103420 2024-04-16T11:41:51.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.611+0000 7ff8dd600700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff8c40692a0 0x7ff8c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:51.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.611+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff8d00871a0 con 0x7ff8e0103420 2024-04-16T11:41:51.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.611+0000 7ff8d7e00700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff8c40692a0 0x7ff8c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:51.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.612+0000 7ff8d7e00700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff8c40692a0 0x7ff8c406b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8004f90 tx=0x7ff8c8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:51.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.615+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff8d0057000 con 0x7ff8e0103420 2024-04-16T11:41:51.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.854+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8e0108a30 con 0x7ff8c40692a0 2024-04-16T11:41:51.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.856+0000 7ff8dd600700 1 -- 172.21.15.44:0/1697793605 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7ff8e0108a30 con 0x7ff8c40692a0 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.858+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff8c40692a0 msgr2=0x7ff8c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.858+0000 7ff8e7800700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff8c40692a0 0x7ff8c406b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8004f90 tx=0x7ff8c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 msgr2=0x7ff8e0113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 0x7ff8e0113f60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c8c0 tx=0x7ff8d00044c0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 shutdown_connections 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff8c40692a0 0x7ff8c406b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff8e0103420 0x7ff8e0113f60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 --2- 172.21.15.44:0/1697793605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8e01046d0 0x7ff8e01144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:51.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 >> 172.21.15.44:0/1697793605 conn(0x7ff8e0072640 msgr2=0x7ff8e0073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 shutdown_connections 2024-04-16T11:41:51.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:51.859+0000 7ff8e7800700 1 -- 172.21.15.44:0/1697793605 wait complete. 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 -- 172.21.15.44:0/952781086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc0708f0 msgr2=0x7f1ffc070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 --2- 172.21.15.44:0/952781086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc0708f0 0x7f1ffc070d60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8004940 tx=0x7f1fe8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 -- 172.21.15.44:0/952781086 shutdown_connections 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 --2- 172.21.15.44:0/952781086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc0708f0 0x7f1ffc070d60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 --2- 172.21.15.44:0/952781086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 -- 172.21.15.44:0/952781086 >> 172.21.15.44:0/952781086 conn(0x7f1ffc0f8580 msgr2=0x7f1ffc0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.011+0000 7f2002e00700 1 -- 172.21.15.44:0/952781086 shutdown_connections 2024-04-16T11:41:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.012+0000 7f2002e00700 1 -- 172.21.15.44:0/952781086 wait complete. 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.012+0000 7f2002e00700 1 Processor -- start 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.012+0000 7f2002e00700 1 -- start start 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f2002e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ffc0708f0 0x7f1ffc114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f2002e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc1145d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f2002e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc114bd0 con 0x7f1ffc071ea0 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f2002e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc115420 con 0x7f1ffc0708f0 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc1145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc1145d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33846/0 (socket says 172.21.15.44:33846) 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f1ffb400700 1 -- 172.21.15.44:0/2007321108 learned_addr learned my addr 172.21.15.44:0/2007321108 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.013+0000 7f1ffbe00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ffc0708f0 0x7f1ffc114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ffb400700 1 -- 172.21.15.44:0/2007321108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ffc0708f0 msgr2=0x7f1ffc114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ffb400700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ffc0708f0 0x7f1ffc114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ffb400700 1 -- 172.21.15.44:0/2007321108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8005040 con 0x7f1ffc071ea0 2024-04-16T11:41:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ffb400700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc1145d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800b3b0 tx=0x7f1fe8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fe800e040 con 0x7f1ffc071ea0 2024-04-16T11:41:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fe8004290 con 0x7f1ffc071ea0 2024-04-16T11:41:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fe801c630 con 0x7f1ffc071ea0 2024-04-16T11:41:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc1156a0 con 0x7f1ffc071ea0 2024-04-16T11:41:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.014+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc115be0 con 0x7f1ffc071ea0 2024-04-16T11:41:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.016+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc05f5f0 con 0x7f1ffc071ea0 2024-04-16T11:41:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.016+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1fe800bbf0 con 0x7f1ffc071ea0 2024-04-16T11:41:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.018+0000 7f1ff3e00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1fe00692b0 0x7f1fe006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.018+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f1fe8088610 con 0x7f1ffc071ea0 2024-04-16T11:41:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.018+0000 7f1ffbe00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1fe00692b0 0x7f1fe006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.019+0000 7f1ffbe00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1fe00692b0 0x7f1fe006b760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1fec007fd0 tx=0x7f1fec009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:52.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.022+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1fe80583b0 con 0x7f1ffc071ea0 2024-04-16T11:41:52.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.250+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ffc115fc0 con 0x7f1fe00692b0 2024-04-16T11:41:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.278+0000 7f1ff3e00700 1 -- 172.21.15.44:0/2007321108 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f1ffc115fc0 con 0x7f1fe00692b0 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (7m) 107s ago 9m 13.5M - 0.25.0 c8568f914cd2 8136acabd663 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (9m) 107s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (8m) 113s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (7m) 107s ago 9m 75.2M - 9.4.7 954c08fa6188 afde3dcbfd10 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (2m) 107s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (2m) 107s ago 2m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (2m) 113s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (2m) 113s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:9283 running (11m) 107s ago 11m 483M - 17.2.7-847-g45317f95 b333581ccdf5 53edec877549 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (8m) 113s ago 8m 432M - 17.2.7-847-g45317f95 b333581ccdf5 f6be9ec13da1 2024-04-16T11:41:52.277 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (11m) 107s ago 11m 43.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (8m) 113s ago 8m 34.6M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (9m) 107s ago 9m 7746k - 1.5.0 0da6a335fe13 dbe9480d7dd0 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (8m) 113s ago 8m 7486k - 1.5.0 0da6a335fe13 e37d89867e75 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (7m) 107s ago 7m 44.5M 4290M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 107s ago 6m 46.9M 4290M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (5m) 107s ago 5m 43.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (5m) 113s ago 5m 42.9M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (4m) 113s ago 4m 40.6M 5314M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (4m) 113s ago 4m 42.6M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:41:52.278 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (7m) 107s ago 8m 44.5M - 2.43.0 a07b618ecd1d 99d239fdb386 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1fe00692b0 msgr2=0x7f1fe006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1fe00692b0 0x7f1fe006b760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1fec007fd0 tx=0x7f1fec009040 comp rx=0 tx=0).stop 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 msgr2=0x7f1ffc1145d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc1145d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800b3b0 tx=0x7f1fe8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 shutdown_connections 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f1fe00692b0 0x7f1fe006b760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ffc0708f0 0x7f1ffc114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 --2- 172.21.15.44:0/2007321108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ffc071ea0 0x7f1ffc1145d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.281+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 >> 172.21.15.44:0/2007321108 conn(0x7f1ffc0f8580 msgr2=0x7f1ffc0f9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.282+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 shutdown_connections 2024-04-16T11:41:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.282+0000 7f2002e00700 1 -- 172.21.15.44:0/2007321108 wait complete. 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.438+0000 7ff312c00700 1 -- 172.21.15.44:0/3770253321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c072820 msgr2=0x7ff30c072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.438+0000 7ff312c00700 1 --2- 172.21.15.44:0/3770253321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c072820 0x7ff30c072c30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc004940 tx=0x7ff2fc009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 -- 172.21.15.44:0/3770253321 shutdown_connections 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 --2- 172.21.15.44:0/3770253321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c073170 0x7ff30c073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 --2- 172.21.15.44:0/3770253321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c072820 0x7ff30c072c30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 -- 172.21.15.44:0/3770253321 >> 172.21.15.44:0/3770253321 conn(0x7ff30c0fa950 msgr2=0x7ff30c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 -- 172.21.15.44:0/3770253321 shutdown_connections 2024-04-16T11:41:52.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 -- 172.21.15.44:0/3770253321 wait complete. 2024-04-16T11:41:52.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.439+0000 7ff312c00700 1 Processor -- start 2024-04-16T11:41:52.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff312c00700 1 -- start start 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff312c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c072820 0x7ff30c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff312c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 0x7ff30c1194d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff312c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c114960 con 0x7ff30c1144a0 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff312c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c114aa0 con 0x7ff30c072820 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff30b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 0x7ff30c1194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff30b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 0x7ff30c1194d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33872/0 (socket says 172.21.15.44:33872) 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff30b400700 1 -- 172.21.15.44:0/2323280482 learned_addr learned my addr 172.21.15.44:0/2323280482 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.440+0000 7ff30be00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c072820 0x7ff30c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff30b400700 1 -- 172.21.15.44:0/2323280482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c072820 msgr2=0x7ff30c113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff30b400700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c072820 0x7ff30c113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff30b400700 1 -- 172.21.15.44:0/2323280482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc005040 con 0x7ff30c1144a0 2024-04-16T11:41:52.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff30be00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c072820 0x7ff30c113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:41:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff30b400700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 0x7ff30c1194d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8002a50 tx=0x7ff2f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f8007cc0 con 0x7ff30c1144a0 2024-04-16T11:41:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2f800f040 con 0x7ff30c1144a0 2024-04-16T11:41:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c119a70 con 0x7ff30c1144a0 2024-04-16T11:41:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.441+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f8009640 con 0x7ff30c1144a0 2024-04-16T11:41:52.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.442+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c119fe0 con 0x7ff30c1144a0 2024-04-16T11:41:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.443+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c04c710 con 0x7ff30c1144a0 2024-04-16T11:41:52.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.444+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff2f80097a0 con 0x7ff30c1144a0 2024-04-16T11:41:52.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.445+0000 7ff303e00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff2f00692a0 0x7ff2f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.445+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff2f8086f00 con 0x7ff30c1144a0 2024-04-16T11:41:52.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.445+0000 7ff30be00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff2f00692a0 0x7ff2f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.446+0000 7ff30be00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff2f00692a0 0x7ff2f006b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00b970 tx=0x7ff2fc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:52.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.452+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff2f8056cc0 con 0x7ff30c1144a0 2024-04-16T11:41:52.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.755+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff30c073350 con 0x7ff30c1144a0 2024-04-16T11:41:52.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.757+0000 7ff303e00700 1 -- 172.21.15.44:0/2323280482 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7ff2f8056850 con 0x7ff30c1144a0 2024-04-16T11:41:52.755 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:41:52.755 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:41:52.755 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:41:52.755 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:52.755 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:41:52.756 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff2f00692a0 msgr2=0x7ff2f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff2f00692a0 0x7ff2f006b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00b970 tx=0x7ff2fc009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 msgr2=0x7ff30c1194d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 0x7ff30c1194d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8002a50 tx=0x7ff2f80046b0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 shutdown_connections 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7ff2f00692a0 0x7ff2f006b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff30c072820 0x7ff30c113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 --2- 172.21.15.44:0/2323280482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff30c1144a0 0x7ff30c1194d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.759+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 >> 172.21.15.44:0/2323280482 conn(0x7ff30c0fa950 msgr2=0x7ff30c0fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.760+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 shutdown_connections 2024-04-16T11:41:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.760+0000 7ff312c00700 1 -- 172.21.15.44:0/2323280482 wait complete. 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.913+0000 7f2fdc800700 1 -- 172.21.15.44:0/16619513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 msgr2=0x7f2fd40fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.913+0000 7f2fdc800700 1 --2- 172.21.15.44:0/16619513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd40fe070 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4004940 tx=0x7f2fc4009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.914+0000 7f2fdc800700 1 -- 172.21.15.44:0/16619513 shutdown_connections 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.914+0000 7f2fdc800700 1 --2- 172.21.15.44:0/16619513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd40fe070 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.914+0000 7f2fdc800700 1 --2- 172.21.15.44:0/16619513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fd40fcdc0 0x7f2fd40fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.914+0000 7f2fdc800700 1 -- 172.21.15.44:0/16619513 >> 172.21.15.44:0/16619513 conn(0x7f2fd40f8540 msgr2=0x7f2fd40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.914+0000 7f2fdc800700 1 -- 172.21.15.44:0/16619513 shutdown_connections 2024-04-16T11:41:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.914+0000 7f2fdc800700 1 -- 172.21.15.44:0/16619513 wait complete. 2024-04-16T11:41:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.915+0000 7f2fdc800700 1 Processor -- start 2024-04-16T11:41:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.915+0000 7f2fdc800700 1 -- start start 2024-04-16T11:41:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.915+0000 7f2fdc800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fd40fcdc0 0x7f2fd410fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.915+0000 7f2fdc800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd4110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.915+0000 7f2fdc800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4110890 con 0x7f2fd40fd710 2024-04-16T11:41:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.915+0000 7f2fdc800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4110fd0 con 0x7f2fd40fcdc0 2024-04-16T11:41:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd4110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd4110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33894/0 (socket says 172.21.15.44:33894) 2024-04-16T11:41:52.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 -- 172.21.15.44:0/1448781911 learned_addr learned my addr 172.21.15.44:0/1448781911 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9e00700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fd40fcdc0 0x7f2fd410fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 -- 172.21.15.44:0/1448781911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fd40fcdc0 msgr2=0x7f2fd410fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fd40fcdc0 0x7f2fd410fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 -- 172.21.15.44:0/1448781911 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4005040 con 0x7f2fd40fd710 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.916+0000 7f2fd9400700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd4110290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400b970 tx=0x7f2fc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.917+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc400e040 con 0x7f2fd40fd710 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.917+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fc400bbf0 con 0x7f2fd40fd710 2024-04-16T11:41:52.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.917+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd4111250 con 0x7f2fd40fd710 2024-04-16T11:41:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.917+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc401c630 con 0x7f2fd40fd710 2024-04-16T11:41:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.917+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd4111810 con 0x7f2fd40fd710 2024-04-16T11:41:52.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.918+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4101a60 con 0x7f2fd40fd710 2024-04-16T11:41:52.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.919+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2fc400bd60 con 0x7f2fd40fd710 2024-04-16T11:41:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.920+0000 7f2fce000700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2fbc0692b0 0x7f2fbc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.920+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2fc4088550 con 0x7f2fd40fd710 2024-04-16T11:41:52.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.920+0000 7f2fd9e00700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2fbc0692b0 0x7f2fbc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:52.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.921+0000 7f2fd9e00700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2fbc0692b0 0x7f2fbc06b760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009fd0 tx=0x7f2fc8009f00 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:52.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:52.925+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2fc40583a0 con 0x7f2fd40fd710 2024-04-16T11:41:53.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.189+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2fd405f5f0 con 0x7f2fd40fd710 2024-04-16T11:41:53.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.189+0000 7f2fce000700 1 -- 172.21.15.44:0/1448781911 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7f2fc4057f30 con 0x7f2fd40fd710 2024-04-16T11:41:53.188 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:41:53.189 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:41:53.190 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:53.191 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:41:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.192+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2fbc0692b0 msgr2=0x7f2fbc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.192+0000 7f2fdc800700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2fbc0692b0 0x7f2fbc06b760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009fd0 tx=0x7f2fc8009f00 comp rx=0 tx=0).stop 2024-04-16T11:41:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.192+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 msgr2=0x7f2fd4110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.192+0000 7f2fdc800700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd4110290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400b970 tx=0x7f2fc4009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:53.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 shutdown_connections 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f2fbc0692b0 0x7f2fbc06b760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fd40fcdc0 0x7f2fd410fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 --2- 172.21.15.44:0/1448781911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd40fd710 0x7f2fd4110290 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 >> 172.21.15.44:0/1448781911 conn(0x7f2fd40f8540 msgr2=0x7f2fd40f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 shutdown_connections 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.193+0000 7f2fdc800700 1 -- 172.21.15.44:0/1448781911 wait complete. 2024-04-16T11:41:53.193 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 -- 172.21.15.44:0/3211446231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 msgr2=0x7f9b280703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 --2- 172.21.15.44:0/3211446231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b280703b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009ce0 tx=0x7f9b1c004d10 comp rx=0 tx=0).stop 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 -- 172.21.15.44:0/3211446231 shutdown_connections 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 --2- 172.21.15.44:0/3211446231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b280708f0 0x7f9b28070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 --2- 172.21.15.44:0/3211446231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b280703b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 -- 172.21.15.44:0/3211446231 >> 172.21.15.44:0/3211446231 conn(0x7f9b280f83a0 msgr2=0x7f9b280fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 -- 172.21.15.44:0/3211446231 shutdown_connections 2024-04-16T11:41:53.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.340+0000 7f9b2e600700 1 -- 172.21.15.44:0/3211446231 wait complete. 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b2e600700 1 Processor -- start 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b2e600700 1 -- start start 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b2e600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b280708f0 0x7f9b2810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b2e600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b28110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b2e600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b28110850 con 0x7f9b28071ea0 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b2e600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b28110f90 con 0x7f9b280708f0 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b27e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b28110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b27e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b28110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33916/0 (socket says 172.21.15.44:33916) 2024-04-16T11:41:53.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.341+0000 7f9b27e00700 1 -- 172.21.15.44:0/1616411526 learned_addr learned my addr 172.21.15.44:0/1616411526 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.342+0000 7f9b2d200700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b280708f0 0x7f9b2810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.342+0000 7f9b27e00700 1 -- 172.21.15.44:0/1616411526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b280708f0 msgr2=0x7f9b2810fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.342+0000 7f9b27e00700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b280708f0 0x7f9b2810fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.342+0000 7f9b27e00700 1 -- 172.21.15.44:0/1616411526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c009940 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.342+0000 7f9b27e00700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b28110250 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9b14002a50 tx=0x7f9b140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.342+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b14007cc0 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.343+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b1400f040 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.343+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b1400da30 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.343+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b28111270 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.343+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b28111830 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.344+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b280fc020 con 0x7f9b28071ea0 2024-04-16T11:41:53.342 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:53 smithi044 ceph-mon[28731]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:53.342 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:53 smithi044 ceph-mon[28731]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:53.343 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:53 smithi044 ceph-mon[28731]: pgmap v291: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:53.343 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:53 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2323280482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:41:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.345+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9b1400db90 con 0x7f9b28071ea0 2024-04-16T11:41:53.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.346+0000 7f9b24c00700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9b100692a0 0x7f9b1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:53.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.346+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9b14086f10 con 0x7f9b28071ea0 2024-04-16T11:41:53.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.346+0000 7f9b2d200700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9b100692a0 0x7f9b1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.347+0000 7f9b2d200700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9b100692a0 0x7f9b1006b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009cb0 tx=0x7f9b1c004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:53.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.350+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9b14056c30 con 0x7f9b28071ea0 2024-04-16T11:41:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:53 smithi045 ceph-mon[36308]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:53 smithi045 ceph-mon[36308]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:53 smithi045 ceph-mon[36308]: pgmap v291: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:53 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2323280482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:41:53.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.584+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 --> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b28111bb0 con 0x7f9b100692a0 2024-04-16T11:41:53.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.587+0000 7f9b24c00700 1 -- 172.21.15.44:0/1616411526 <== mgr.14212 v2:172.21.15.44:6800/2086828317 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f9b28111bb0 con 0x7f9b100692a0 2024-04-16T11:41:53.585 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:41:53.585 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:41:53.585 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:41:53.585 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:41:53.585 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:41:53.586 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "0/21 daemons upgraded", 2024-04-16T11:41:53.586 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa image on host smithi045", 2024-04-16T11:41:53.586 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:41:53.586 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:41:53.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9b100692a0 msgr2=0x7f9b1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9b100692a0 0x7f9b1006b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009cb0 tx=0x7f9b1c004d10 comp rx=0 tx=0).stop 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 msgr2=0x7f9b28110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b28110250 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9b14002a50 tx=0x7f9b140046b0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 shutdown_connections 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f9b100692a0 0x7f9b1006b750 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b280708f0 0x7f9b2810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 --2- 172.21.15.44:0/1616411526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b28071ea0 0x7f9b28110250 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 >> 172.21.15.44:0/1616411526 conn(0x7f9b280f83a0 msgr2=0x7f9b280f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 shutdown_connections 2024-04-16T11:41:53.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.590+0000 7f9b2e600700 1 -- 172.21.15.44:0/1616411526 wait complete. 2024-04-16T11:41:53.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 -- 172.21.15.44:0/4091384061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 msgr2=0x7f02ec070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 --2- 172.21.15.44:0/4091384061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec070d00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f02d8004940 tx=0x7f02d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:53.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 -- 172.21.15.44:0/4091384061 shutdown_connections 2024-04-16T11:41:53.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 --2- 172.21.15.44:0/4091384061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec070d00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 --2- 172.21.15.44:0/4091384061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec071e40 0x7f02ec070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 -- 172.21.15.44:0/4091384061 >> 172.21.15.44:0/4091384061 conn(0x7f02ec0f83a0 msgr2=0x7f02ec0fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 -- 172.21.15.44:0/4091384061 shutdown_connections 2024-04-16T11:41:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.750+0000 7f02ebe00700 1 -- 172.21.15.44:0/4091384061 wait complete. 2024-04-16T11:41:53.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.751+0000 7f02ebe00700 1 Processor -- start 2024-04-16T11:41:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.751+0000 7f02ebe00700 1 -- start start 2024-04-16T11:41:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02ebe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec10fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02ebe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec071e40 0x7f02ec10ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02ebe00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec110590 con 0x7f02ec070890 2024-04-16T11:41:53.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02ebe00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec110de0 con 0x7f02ec071e40 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02eaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec10fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02eaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec10fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33922/0 (socket says 172.21.15.44:33922) 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02eaa00700 1 -- 172.21.15.44:0/3319673273 learned_addr learned my addr 172.21.15.44:0/3319673273 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.752+0000 7f02ea000700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec071e40 0x7f02ec10ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02eaa00700 1 -- 172.21.15.44:0/3319673273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec071e40 msgr2=0x7f02ec10ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02eaa00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec071e40 0x7f02ec10ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02eaa00700 1 -- 172.21.15.44:0/3319673273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8005040 con 0x7f02ec070890 2024-04-16T11:41:53.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02eaa00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec10fa50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00c8e0 tx=0x7f02dc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:53.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02dc00a940 con 0x7f02ec070890 2024-04-16T11:41:53.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02dc00ce80 con 0x7f02ec070890 2024-04-16T11:41:53.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ec1110c0 con 0x7f02ec070890 2024-04-16T11:41:53.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.753+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02dc0113a0 con 0x7f02ec070890 2024-04-16T11:41:53.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.754+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ec111600 con 0x7f02ec070890 2024-04-16T11:41:53.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.754+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ec0fc020 con 0x7f02ec070890 2024-04-16T11:41:53.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.756+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f02dc00aaa0 con 0x7f02ec070890 2024-04-16T11:41:53.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.757+0000 7f02e2a00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f02d00692a0 0x7f02d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:41:53.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.757+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f02dc0872d0 con 0x7f02ec070890 2024-04-16T11:41:53.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.757+0000 7f02ea000700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f02d00692a0 0x7f02d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:41:53.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.758+0000 7f02ea000700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f02d00692a0 0x7f02d006b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02d800b3b0 tx=0x7f02d8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:41:53.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:53.761+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f02dc056ff0 con 0x7f02ec070890 2024-04-16T11:41:54.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.057+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f02ec111970 con 0x7f02ec070890 2024-04-16T11:41:54.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.058+0000 7f02e2a00700 1 -- 172.21.15.44:0/3319673273 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f02dc056b80 con 0x7f02ec070890 2024-04-16T11:41:54.056 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:41:54.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f02d00692a0 msgr2=0x7f02d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:54.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f02d00692a0 0x7f02d006b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02d800b3b0 tx=0x7f02d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:41:54.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 msgr2=0x7f02ec10fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec10fa50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00c8e0 tx=0x7f02dc0044c0 comp rx=0 tx=0).stop 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 shutdown_connections 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:6800/2086828317,v1:172.21.15.44:6801/2086828317] conn(0x7f02d00692a0 0x7f02d006b750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02ec070890 0x7f02ec10fa50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 --2- 172.21.15.44:0/3319673273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02ec071e40 0x7f02ec10ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 >> 172.21.15.44:0/3319673273 conn(0x7f02ec0f83a0 msgr2=0x7f02ec0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.061+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 shutdown_connections 2024-04-16T11:41:54.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:41:54.062+0000 7f02ebe00700 1 -- 172.21.15.44:0/3319673273 wait complete. 2024-04-16T11:41:54.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:54 smithi044 ceph-mon[28731]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:54.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:54 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1448781911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:41:54.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:54 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3319673273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:41:54.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:54 smithi045 ceph-mon[36308]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:54.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:54 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1448781911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:41:54.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:54 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3319673273' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:41:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:55 smithi044 ceph-mon[28731]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:55 smithi044 ceph-mon[28731]: pgmap v292: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:55 smithi045 ceph-mon[36308]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:41:55.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:55 smithi045 ceph-mon[36308]: pgmap v292: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:57 smithi044 ceph-mon[28731]: pgmap v293: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:57 smithi045 ceph-mon[36308]: pgmap v293: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:59.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:41:59 smithi044 ceph-mon[28731]: pgmap v294: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:41:59.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:41:59 smithi045 ceph-mon[36308]: pgmap v294: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: pgmap v295: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: Upgrade: Updating mgr.smithi045.sdlwif 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: Deploying daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:42:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:01 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: pgmap v295: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: Upgrade: Updating mgr.smithi045.sdlwif 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: Deploying daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:42:01.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:01 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:03.378 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:03 smithi045 ceph-mon[36308]: pgmap v296: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:03.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:03 smithi044 ceph-mon[28731]: pgmap v296: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:05.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:05 smithi045 ceph-mon[36308]: pgmap v297: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:05.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:05 smithi044 ceph-mon[28731]: pgmap v297: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:06.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:42:06.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:06 smithi045 ceph-mon[36308]: pgmap v298: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:06.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:06 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:42:06.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:06 smithi044 ceph-mon[28731]: pgmap v298: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:06.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:06.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:06 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:09 smithi045 ceph-mon[36308]: pgmap v299: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:09.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:09 smithi044 ceph-mon[28731]: pgmap v299: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:10.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:10 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:10.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:10 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:10.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:10 smithi045 ceph-mon[36308]: pgmap v300: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:10.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:10 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:10.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:10 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:10.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:10 smithi044 ceph-mon[28731]: pgmap v300: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:12.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:12 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:12.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:12 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:12.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:12 smithi045 ceph-mon[36308]: pgmap v301: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:12.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:12 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:12.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:12 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:12.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:12 smithi044 ceph-mon[28731]: pgmap v301: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:14.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63130 2024-04-16T11:42:14.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 12, success, outsize: 120 2024-04-16T11:42:14.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63208 2024-04-16T11:42:14.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 14, success, outsize: 120 2024-04-16T11:42:14.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63289 2024-04-16T11:42:14.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 16, success, outsize: 120 2024-04-16T11:42:14.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63376 2024-04-16T11:42:14.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 18, success, outsize: 120 2024-04-16T11:42:14.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63460 2024-04-16T11:42:14.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 20, success, outsize: 120 2024-04-16T11:42:14.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 22, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63538 2024-04-16T11:42:14.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 22, success, outsize: 120 2024-04-16T11:42:14.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 24, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63619 2024-04-16T11:42:14.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 24, success, outsize: 120 2024-04-16T11:42:14.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63706 2024-04-16T11:42:14.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 26, success, outsize: 120 2024-04-16T11:42:15.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 28, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63791 2024-04-16T11:42:15.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 28, success, outsize: 120 2024-04-16T11:42:15.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63870 2024-04-16T11:42:15.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 30, success, outsize: 120 2024-04-16T11:42:15.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 32, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63951 2024-04-16T11:42:15.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 32, success, outsize: 120 2024-04-16T11:42:15.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 34, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64038 2024-04-16T11:42:15.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 34, success, outsize: 120 2024-04-16T11:42:15.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64122 2024-04-16T11:42:15.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 36, success, outsize: 120 2024-04-16T11:42:15.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 38, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64200 2024-04-16T11:42:15.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 38, success, outsize: 120 2024-04-16T11:42:15.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64281 2024-04-16T11:42:15.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 40, success, outsize: 120 2024-04-16T11:42:15.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64370 2024-04-16T11:42:15.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 42, success, outsize: 120 2024-04-16T11:42:15.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 44, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64465 2024-04-16T11:42:15.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 44, success, outsize: 120 2024-04-16T11:42:15.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 46, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64543 2024-04-16T11:42:15.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 46, success, outsize: 120 2024-04-16T11:42:15.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64624 2024-04-16T11:42:15.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 48, success, outsize: 120 2024-04-16T11:42:15.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 50, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64711 2024-04-16T11:42:15.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 50, success, outsize: 120 2024-04-16T11:42:15.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 52, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64807 2024-04-16T11:42:15.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 52, success, outsize: 120 2024-04-16T11:42:15.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 54, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64885 2024-04-16T11:42:15.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 54, success, outsize: 120 2024-04-16T11:42:15.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 56, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64966 2024-04-16T11:42:15.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 56, success, outsize: 120 2024-04-16T11:42:15.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 58, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65053 2024-04-16T11:42:15.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 58, success, outsize: 120 2024-04-16T11:42:15.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 60, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65137 2024-04-16T11:42:15.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 60, success, outsize: 120 2024-04-16T11:42:15.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 62, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65215 2024-04-16T11:42:15.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 62, success, outsize: 120 2024-04-16T11:42:15.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 64, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65296 2024-04-16T11:42:15.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 64, success, outsize: 120 2024-04-16T11:42:15.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 66, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65383 2024-04-16T11:42:15.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 66, success, outsize: 120 2024-04-16T11:42:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:15 smithi045 ceph-mon[36308]: pgmap v302: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:15.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:15 smithi044 ceph-mon[28731]: pgmap v302: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif restarted 2024-04-16T11:42:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:42:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:42:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:42:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:42:16.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:42:16.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:16 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: Standby manager daemon smithi045.sdlwif restarted 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:42:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:16 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:17.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: pgmap v303: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: mgrmap e20: smithi044.sczobv(active, since 9m), standbys: smithi045.sdlwif 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr fail", "who": "smithi044.sczobv"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: Activating manager daemon smithi045.sdlwif 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "mgr fail", "who": "smithi044.sczobv"}]': finished 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: mgrmap e21: smithi045.sdlwif(active, starting, since 0.0180165s) 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:42:17.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: Manager daemon smithi045.sdlwif is now available 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:42:17.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/trash_purge_schedule"}]: dispatch 2024-04-16T11:42:17.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:17 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/trash_purge_schedule"}]: dispatch 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: pgmap v303: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: mgrmap e20: smithi044.sczobv(active, since 9m), standbys: smithi045.sdlwif 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:42:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr fail", "who": "smithi044.sczobv"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: Activating manager daemon smithi045.sdlwif 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.14212 172.21.15.44:0/479834426' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "mgr fail", "who": "smithi044.sczobv"}]': finished 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: mgrmap e21: smithi045.sdlwif(active, starting, since 0.0180165s) 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:42:17.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: Manager daemon smithi045.sdlwif is now available 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:42:17.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/trash_purge_schedule"}]: dispatch 2024-04-16T11:42:17.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:17 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.sdlwif/trash_purge_schedule"}]: dispatch 2024-04-16T11:42:18.399 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:18 smithi044 ceph-mon[28731]: Deploying cephadm binary to smithi045 2024-04-16T11:42:18.399 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:18 smithi044 ceph-mon[28731]: Deploying cephadm binary to smithi044 2024-04-16T11:42:18.399 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:18 smithi044 ceph-mon[28731]: mgrmap e22: smithi045.sdlwif(active, since 1.01423s) 2024-04-16T11:42:18.399 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:18 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:18.399 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:18 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:18.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:18 smithi045 ceph-mon[36308]: Deploying cephadm binary to smithi045 2024-04-16T11:42:18.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:18 smithi045 ceph-mon[36308]: Deploying cephadm binary to smithi044 2024-04-16T11:42:18.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:18 smithi045 ceph-mon[36308]: mgrmap e22: smithi045.sdlwif(active, since 1.01423s) 2024-04-16T11:42:18.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:18 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:18.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:18 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:19 smithi045 ceph-mon[36308]: pgmap v3: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:19 smithi045 ceph-mon[36308]: [16/Apr/2024:11:42:18] ENGINE Bus STARTING 2024-04-16T11:42:19.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:19 smithi045 ceph-mon[36308]: [16/Apr/2024:11:42:18] ENGINE Serving on http://172.21.15.45:8765 2024-04-16T11:42:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:19 smithi044 ceph-mon[28731]: pgmap v3: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:19 smithi044 ceph-mon[28731]: [16/Apr/2024:11:42:18] ENGINE Bus STARTING 2024-04-16T11:42:19.408 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:19 smithi044 ceph-mon[28731]: [16/Apr/2024:11:42:18] ENGINE Serving on http://172.21.15.45:8765 2024-04-16T11:42:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:20 smithi044 ceph-mon[28731]: pgmap v4: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:20 smithi044 ceph-mon[28731]: [16/Apr/2024:11:42:18] ENGINE Serving on https://172.21.15.45:7150 2024-04-16T11:42:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:20 smithi044 ceph-mon[28731]: [16/Apr/2024:11:42:18] ENGINE Bus STARTED 2024-04-16T11:42:20.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:20 smithi045 ceph-mon[36308]: pgmap v4: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:20.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:20 smithi045 ceph-mon[36308]: [16/Apr/2024:11:42:18] ENGINE Serving on https://172.21.15.45:7150 2024-04-16T11:42:20.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:20 smithi045 ceph-mon[36308]: [16/Apr/2024:11:42:18] ENGINE Bus STARTED 2024-04-16T11:42:21.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:21 smithi044 ceph-mon[28731]: mgrmap e23: smithi045.sdlwif(active, since 3s) 2024-04-16T11:42:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:21 smithi045 ceph-mon[36308]: mgrmap e23: smithi045.sdlwif(active, since 3s) 2024-04-16T11:42:22.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:22 smithi044 ceph-mon[28731]: pgmap v5: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:22.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:22 smithi045 ceph-mon[36308]: pgmap v5: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:24.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 -- 172.21.15.44:0/1236621946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c071ef0 msgr2=0x7f278c070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 --2- 172.21.15.44:0/1236621946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c071ef0 0x7f278c070400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f277c004970 tx=0x7f277c009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 -- 172.21.15.44:0/1236621946 shutdown_connections 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 --2- 172.21.15.44:0/1236621946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c070940 0x7f278c070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 --2- 172.21.15.44:0/1236621946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c071ef0 0x7f278c070400 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 -- 172.21.15.44:0/1236621946 >> 172.21.15.44:0/1236621946 conn(0x7f278c0f8530 msgr2=0x7f278c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 -- 172.21.15.44:0/1236621946 shutdown_connections 2024-04-16T11:42:24.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.220+0000 7f2792600700 1 -- 172.21.15.44:0/1236621946 wait complete. 2024-04-16T11:42:24.219 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:24 smithi044 ceph-mon[28731]: pgmap v6: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-16T11:42:24.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.221+0000 7f2792600700 1 Processor -- start 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.221+0000 7f2792600700 1 -- start start 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f2792600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 0x7f278c113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f2792600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c071ef0 0x7f278c114520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f2792600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c114b20 con 0x7f278c070940 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f2792600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c115370 con 0x7f278c071ef0 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f278b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 0x7f278c113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f278b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 0x7f278c113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44668/0 (socket says 172.21.15.44:44668) 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.222+0000 7f278b400700 1 -- 172.21.15.44:0/349508062 learned_addr learned my addr 172.21.15.44:0/349508062 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f278aa00700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c071ef0 0x7f278c114520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f278b400700 1 -- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c071ef0 msgr2=0x7f278c114520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f278b400700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c071ef0 0x7f278c114520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f278b400700 1 -- 172.21.15.44:0/349508062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c005040 con 0x7f278c070940 2024-04-16T11:42:24.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f278aa00700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c071ef0 0x7f278c114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:42:24.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f278b400700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 0x7f278c113fe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f277c00b930 tx=0x7f277c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:24.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c00e040 con 0x7f278c070940 2024-04-16T11:42:24.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f277c00bce0 con 0x7f278c070940 2024-04-16T11:42:24.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.223+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c1155f0 con 0x7f278c070940 2024-04-16T11:42:24.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.224+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c01c630 con 0x7f278c070940 2024-04-16T11:42:24.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.224+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c115b30 con 0x7f278c070940 2024-04-16T11:42:24.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.225+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c101180 con 0x7f278c070940 2024-04-16T11:42:24.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.225+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) v1 ==== 46566+0+0 (secure 0 0 0) 0x7f277c013a80 con 0x7f278c070940 2024-04-16T11:42:24.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.225+0000 7f2783400700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2770039470 0x7f277003b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:24.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.225+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f277c04e500 con 0x7f278c070940 2024-04-16T11:42:24.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.226+0000 7f278aa00700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2770039470 0x7f277003b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:24.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.227+0000 7f278aa00700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2770039470 0x7f277003b920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2778007fd0 tx=0x7f2778009040 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:24.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.232+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f277c011960 con 0x7f278c070940 2024-04-16T11:42:24.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.493+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f278c115e10 con 0x7f2770039470 2024-04-16T11:42:24.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.497+0000 7f2783400700 1 -- 172.21.15.44:0/349508062 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f278c115e10 con 0x7f2770039470 2024-04-16T11:42:24.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2770039470 msgr2=0x7f277003b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2770039470 0x7f277003b920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2778007fd0 tx=0x7f2778009040 comp rx=0 tx=0).stop 2024-04-16T11:42:24.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 msgr2=0x7f278c113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 0x7f278c113fe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f277c00b930 tx=0x7f277c009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 shutdown_connections 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2770039470 0x7f277003b920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f278c070940 0x7f278c113fe0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 --2- 172.21.15.44:0/349508062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f278c071ef0 0x7f278c114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 >> 172.21.15.44:0/349508062 conn(0x7f278c0f8530 msgr2=0x7f278c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 shutdown_connections 2024-04-16T11:42:24.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.500+0000 7f2792600700 1 -- 172.21.15.44:0/349508062 wait complete. 2024-04-16T11:42:24.514 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:42:24.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:24 smithi045 ceph-mon[36308]: pgmap v6: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 -- 172.21.15.44:0/397864931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800662f0 msgr2=0x7fc0800ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 --2- 172.21.15.44:0/397864931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800662f0 0x7fc0800ffa30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc070004940 tx=0x7fc070009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 -- 172.21.15.44:0/397864931 shutdown_connections 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 --2- 172.21.15.44:0/397864931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0800fff70 0x7fc080102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 --2- 172.21.15.44:0/397864931 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800662f0 0x7fc0800ffa30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 -- 172.21.15.44:0/397864931 >> 172.21.15.44:0/397864931 conn(0x7fc0800f74c0 msgr2=0x7fc0800f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 -- 172.21.15.44:0/397864931 shutdown_connections 2024-04-16T11:42:24.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.659+0000 7fc085400700 1 -- 172.21.15.44:0/397864931 wait complete. 2024-04-16T11:42:24.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.660+0000 7fc085400700 1 Processor -- start 2024-04-16T11:42:24.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.660+0000 7fc085400700 1 -- start start 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.660+0000 7fc085400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0800662f0 0x7fc0800fd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.660+0000 7fc085400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 0x7fc0800fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc085400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc080111790 con 0x7fc0800fff70 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc085400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0801118d0 con 0x7fc0800662f0 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 0x7fc0800fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 0x7fc0800fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44672/0 (socket says 172.21.15.44:44672) 2024-04-16T11:42:24.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07de00700 1 -- 172.21.15.44:0/3865223943 learned_addr learned my addr 172.21.15.44:0/3865223943 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07e800700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0800662f0 0x7fc0800fd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07de00700 1 -- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0800662f0 msgr2=0x7fc0800fd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07de00700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0800662f0 0x7fc0800fd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.661+0000 7fc07de00700 1 -- 172.21.15.44:0/3865223943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc070005040 con 0x7fc0800fff70 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.662+0000 7fc07de00700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 0x7fc0800fb930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00b7c0 tx=0x7fc06c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.662+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06c00f6f0 con 0x7fc0800fff70 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.662+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc06c00fd20 con 0x7fc0800fff70 2024-04-16T11:42:24.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.662+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06c0103a0 con 0x7fc0800fff70 2024-04-16T11:42:24.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.662+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0800fbf90 con 0x7fc0800fff70 2024-04-16T11:42:24.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.662+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0800fc500 con 0x7fc0800fff70 2024-04-16T11:42:24.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.663+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08005f5f0 con 0x7fc0800fff70 2024-04-16T11:42:24.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.664+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) v1 ==== 46566+0+0 (secure 0 0 0) 0x7fc06c011400 con 0x7fc0800fff70 2024-04-16T11:42:24.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.664+0000 7fc076a00700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc064039470 0x7fc06403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:24.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.664+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fc06c04cfb0 con 0x7fc0800fff70 2024-04-16T11:42:24.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.665+0000 7fc07e800700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc064039470 0x7fc06403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:24.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.666+0000 7fc07e800700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc064039470 0x7fc06403b920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc070004890 tx=0x7fc070009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:24.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.670+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fc06c081930 con 0x7fc0800fff70 2024-04-16T11:42:24.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.931+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0800f8fb0 con 0x7fc064039470 2024-04-16T11:42:24.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.935+0000 7fc076a00700 1 -- 172.21.15.44:0/3865223943 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fc0800f8fb0 con 0x7fc064039470 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc064039470 msgr2=0x7fc06403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc064039470 0x7fc06403b920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc070004890 tx=0x7fc070009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 msgr2=0x7fc0800fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 0x7fc0800fb930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00b7c0 tx=0x7fc06c009250 comp rx=0 tx=0).stop 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 shutdown_connections 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc064039470 0x7fc06403b920 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0800662f0 0x7fc0800fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 --2- 172.21.15.44:0/3865223943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0800fff70 0x7fc0800fb930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.937+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 >> 172.21.15.44:0/3865223943 conn(0x7fc0800f74c0 msgr2=0x7fc0800f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.938+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 shutdown_connections 2024-04-16T11:42:24.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:24.938+0000 7fc085400700 1 -- 172.21.15.44:0/3865223943 wait complete. 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 -- 172.21.15.44:0/3904500231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fce40 msgr2=0x7f2acc0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 --2- 172.21.15.44:0/3904500231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fce40 0x7f2acc0fd250 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0004940 tx=0x7f2ac0009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 -- 172.21.15.44:0/3904500231 shutdown_connections 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 --2- 172.21.15.44:0/3904500231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2acc0fe040 0x7f2acc0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 --2- 172.21.15.44:0/3904500231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fce40 0x7f2acc0fd250 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 -- 172.21.15.44:0/3904500231 >> 172.21.15.44:0/3904500231 conn(0x7f2acc0f8530 msgr2=0x7f2acc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 -- 172.21.15.44:0/3904500231 shutdown_connections 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.089+0000 7f2ad3400700 1 -- 172.21.15.44:0/3904500231 wait complete. 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.090+0000 7f2ad3400700 1 Processor -- start 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2ad3400700 1 -- start start 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2ad3400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2acc0fce40 0x7f2acc10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2ad3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 0x7f2acc110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:25.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2ad3400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc110f80 con 0x7f2acc0fe040 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2ad3400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1110f0 con 0x7f2acc0fce40 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2acbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 0x7f2acc110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2acbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 0x7f2acc110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44678/0 (socket says 172.21.15.44:44678) 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.091+0000 7f2acbe00700 1 -- 172.21.15.44:0/2218168885 learned_addr learned my addr 172.21.15.44:0/2218168885 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.092+0000 7f2ad0a00700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2acc0fce40 0x7f2acc10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.092+0000 7f2acbe00700 1 -- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2acc0fce40 msgr2=0x7f2acc10fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.092+0000 7f2acbe00700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2acc0fce40 0x7f2acc10fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.092+0000 7f2acbe00700 1 -- 172.21.15.44:0/2218168885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0005040 con 0x7f2acc0fe040 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.092+0000 7f2acbe00700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 0x7f2acc110270 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800b790 tx=0x7f2ab8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.093+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ab800f770 con 0x7f2acc0fe040 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.093+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ab800fda0 con 0x7f2acc0fe040 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.093+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ab800d310 con 0x7f2acc0fe040 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.093+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc1113d0 con 0x7f2acc0fe040 2024-04-16T11:42:25.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.093+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc111940 con 0x7f2acc0fe040 2024-04-16T11:42:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.094+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc1019d0 con 0x7f2acc0fe040 2024-04-16T11:42:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.094+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) v1 ==== 46566+0+0 (secure 0 0 0) 0x7f2ab8010420 con 0x7f2acc0fe040 2024-04-16T11:42:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.095+0000 7f2ac8c00700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2ab4039480 0x7f2ab403b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.095+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f2ab804d1f0 con 0x7f2acc0fe040 2024-04-16T11:42:25.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.096+0000 7f2ad0a00700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2ab4039480 0x7f2ab403b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:25.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.097+0000 7f2ad0a00700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2ab4039480 0x7f2ab403b930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b3b0 tx=0x7f2ac0009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:25.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.101+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f2ab800e2e0 con 0x7f2acc0fe040 2024-04-16T11:42:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.343+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2acc111de0 con 0x7f2ab4039480 2024-04-16T11:42:25.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.375+0000 7f2ac8c00700 1 -- 172.21.15.44:0/2218168885 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f2acc111de0 con 0x7f2ab4039480 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (8m) 2m ago 9m 13.5M - 0.25.0 c8568f914cd2 8136acabd663 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (10m) 2m ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (8m) 15s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (8m) 2m ago 9m 75.2M - 9.4.7 954c08fa6188 afde3dcbfd10 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (2m) 2m ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (2m) 2m ago 2m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (2m) 15s ago 2m 11.2M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (2m) 15s ago 2m 11.2M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:9283 running (11m) 2m ago 11m 483M - 17.2.7-847-g45317f95 b333581ccdf5 53edec877549 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (20s) 15s ago 8m 53.9M - 19.0.0-2884-g218a6914 a4e4811815e4 de1d7adc1838 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (11m) 2m ago 12m 43.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:42:25.375 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (8m) 15s ago 8m 35.6M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (10m) 2m ago 10m 7746k - 1.5.0 0da6a335fe13 dbe9480d7dd0 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (8m) 15s ago 8m 7629k - 1.5.0 0da6a335fe13 e37d89867e75 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (7m) 2m ago 7m 44.5M 4290M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 2m ago 6m 46.9M 4290M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (6m) 2m ago 6m 43.4M 4290M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (5m) 15s ago 5m 43.9M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (5m) 15s ago 5m 42.1M 5314M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (4m) 15s ago 4m 44.1M 5314M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:42:25.376 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (7m) 2m ago 9m 44.5M - 2.43.0 a07b618ecd1d 99d239fdb386 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.378+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2ab4039480 msgr2=0x7f2ab403b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.378+0000 7f2ad3400700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2ab4039480 0x7f2ab403b930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b3b0 tx=0x7f2ac0009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.378+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 msgr2=0x7f2acc110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.378+0000 7f2ad3400700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 0x7f2acc110270 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800b790 tx=0x7f2ab8009250 comp rx=0 tx=0).stop 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 shutdown_connections 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2ab4039480 0x7f2ab403b930 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2acc0fce40 0x7f2acc10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 --2- 172.21.15.44:0/2218168885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2acc0fe040 0x7f2acc110270 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 >> 172.21.15.44:0/2218168885 conn(0x7f2acc0f8530 msgr2=0x7f2acc0f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 shutdown_connections 2024-04-16T11:42:25.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.379+0000 7f2ad3400700 1 -- 172.21.15.44:0/2218168885 wait complete. 2024-04-16T11:42:25.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 -- 172.21.15.44:0/223793628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd80709a0 msgr2=0x7f2dd8070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 --2- 172.21.15.44:0/223793628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd80709a0 0x7f2dd8070e10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4004940 tx=0x7f2dc4009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:25.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 -- 172.21.15.44:0/223793628 shutdown_connections 2024-04-16T11:42:25.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 --2- 172.21.15.44:0/223793628 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd80709a0 0x7f2dd8070e10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 --2- 172.21.15.44:0/223793628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd8071f50 0x7f2dd8070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 -- 172.21.15.44:0/223793628 >> 172.21.15.44:0/223793628 conn(0x7f2dd80f8530 msgr2=0x7f2dd80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:25.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 -- 172.21.15.44:0/223793628 shutdown_connections 2024-04-16T11:42:25.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.527+0000 7f2ddf800700 1 -- 172.21.15.44:0/223793628 wait complete. 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.528+0000 7f2ddf800700 1 Processor -- start 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.528+0000 7f2ddf800700 1 -- start start 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2ddf800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd80709a0 0x7f2dd810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2ddf800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 0x7f2dd8110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2ddf800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd8110780 con 0x7f2dd8071f50 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2ddf800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd8110fd0 con 0x7f2dd80709a0 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2dd7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 0x7f2dd8110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2dd7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 0x7f2dd8110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44694/0 (socket says 172.21.15.44:44694) 2024-04-16T11:42:25.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2dd7e00700 1 -- 172.21.15.44:0/3890512157 learned_addr learned my addr 172.21.15.44:0/3890512157 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2ddce00700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd80709a0 0x7f2dd810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2dd7e00700 1 -- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd80709a0 msgr2=0x7f2dd810fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2dd7e00700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd80709a0 0x7f2dd810fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.529+0000 7f2dd7e00700 1 -- 172.21.15.44:0/3890512157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4005040 con 0x7f2dd8071f50 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.530+0000 7f2dd7e00700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 0x7f2dd8110180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009150 tx=0x7f2dc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.530+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc400e040 con 0x7f2dd8071f50 2024-04-16T11:42:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.530+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dc400bbf0 con 0x7f2dd8071f50 2024-04-16T11:42:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.530+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd8111250 con 0x7f2dd8071f50 2024-04-16T11:42:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.530+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd8111790 con 0x7f2dd8071f50 2024-04-16T11:42:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.531+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc401c630 con 0x7f2dd8071f50 2024-04-16T11:42:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.531+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd805f5f0 con 0x7f2dd8071f50 2024-04-16T11:42:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.532+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) v1 ==== 46566+0+0 (secure 0 0 0) 0x7f2dc400bd60 con 0x7f2dd8071f50 2024-04-16T11:42:25.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.533+0000 7f2dd4c00700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2dc0039480 0x7f2dc003b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.533+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f2dc404e700 con 0x7f2dd8071f50 2024-04-16T11:42:25.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.534+0000 7f2ddce00700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2dc0039480 0x7f2dc003b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:25.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.535+0000 7f2ddce00700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2dc0039480 0x7f2dc003b930 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009fd0 tx=0x7f2dcc009f00 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:25.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.538+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f2dc4011960 con 0x7f2dd8071f50 2024-04-16T11:42:25.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.893+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2dd8107b90 con 0x7f2dd8071f50 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.894+0000 7f2dd4c00700 1 -- 172.21.15.44:0/3890512157 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f2dc4023560 con 0x7f2dd8071f50 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:42:25.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:42:25.893 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:42:25.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.896+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2dc0039480 msgr2=0x7f2dc003b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.896+0000 7f2ddf800700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2dc0039480 0x7f2dc003b930 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009fd0 tx=0x7f2dcc009f00 comp rx=0 tx=0).stop 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.896+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 msgr2=0x7f2dd8110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 0x7f2dd8110180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009150 tx=0x7f2dc4009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 shutdown_connections 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f2dc0039480 0x7f2dc003b930 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dd80709a0 0x7f2dd810fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 --2- 172.21.15.44:0/3890512157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd8071f50 0x7f2dd8110180 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 >> 172.21.15.44:0/3890512157 conn(0x7f2dd80f8530 msgr2=0x7f2dd80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:25.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 shutdown_connections 2024-04-16T11:42:25.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:25.897+0000 7f2ddf800700 1 -- 172.21.15.44:0/3890512157 wait complete. 2024-04-16T11:42:26.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 -- 172.21.15.44:0/2719186702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fcdc0 msgr2=0x7f55780fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 --2- 172.21.15.44:0/2719186702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fcdc0 0x7f55780fd230 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5564004940 tx=0x7f5564009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 -- 172.21.15.44:0/2719186702 shutdown_connections 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 --2- 172.21.15.44:0/2719186702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fcdc0 0x7f55780fd230 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 --2- 172.21.15.44:0/2719186702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fbb10 0x7f55780fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 -- 172.21.15.44:0/2719186702 >> 172.21.15.44:0/2719186702 conn(0x7f55780f7270 msgr2=0x7f55780f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 -- 172.21.15.44:0/2719186702 shutdown_connections 2024-04-16T11:42:26.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.045+0000 7f557ce00700 1 -- 172.21.15.44:0/2719186702 wait complete. 2024-04-16T11:42:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.046+0000 7f557ce00700 1 Processor -- start 2024-04-16T11:42:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.046+0000 7f557ce00700 1 -- start start 2024-04-16T11:42:26.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f557ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 0x7f5578112990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f557ce00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fcdc0 0x7f5578114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f557ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5578066250 con 0x7f55780fbb10 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f557ce00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5578066390 con 0x7f55780fcdc0 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f5577400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 0x7f5578112990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f5577400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 0x7f5578112990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44714/0 (socket says 172.21.15.44:44714) 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f5577400700 1 -- 172.21.15.44:0/4090473303 learned_addr learned my addr 172.21.15.44:0/4090473303 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.047+0000 7f5576a00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fcdc0 0x7f5578114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f5577400700 1 -- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fcdc0 msgr2=0x7f5578114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f5577400700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fcdc0 0x7f5578114ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f5577400700 1 -- 172.21.15.44:0/4090473303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5564005040 con 0x7f55780fbb10 2024-04-16T11:42:26.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f5576a00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fcdc0 0x7f5578114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:42:26.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f5577400700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 0x7f5578112990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f556800c8c0 tx=0x7f55680044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:26.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f556800a980 con 0x7f55780fbb10 2024-04-16T11:42:26.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f556800ce60 con 0x7f55780fbb10 2024-04-16T11:42:26.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55680115e0 con 0x7f55780fbb10 2024-04-16T11:42:26.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578115420 con 0x7f55780fbb10 2024-04-16T11:42:26.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.048+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5578115840 con 0x7f55780fbb10 2024-04-16T11:42:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.050+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557805f5f0 con 0x7f55780fbb10 2024-04-16T11:42:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.050+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) v1 ==== 46566+0+0 (secure 0 0 0) 0x7f5568012400 con 0x7f55780fbb10 2024-04-16T11:42:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.050+0000 7f556f400700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f555c039480 0x7f555c03b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.050+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f556804d250 con 0x7f55780fbb10 2024-04-16T11:42:26.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.051+0000 7f5576a00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f555c039480 0x7f555c03b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.052+0000 7f5576a00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f555c039480 0x7f555c03b930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f556400b7a0 tx=0x7f5564009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:26.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.055+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f5568012870 con 0x7f55780fbb10 2024-04-16T11:42:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.333+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5578104600 con 0x7f55780fbb10 2024-04-16T11:42:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.335+0000 7f556f400700 1 -- 172.21.15.44:0/4090473303 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7f55680126b0 con 0x7f55780fbb10 2024-04-16T11:42:26.333 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:42:26.333 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:42:26.333 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:42:26.334 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:42:26.335 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:26.336 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.337+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f555c039480 msgr2=0x7f555c03b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.337+0000 7f557ce00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f555c039480 0x7f555c03b930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f556400b7a0 tx=0x7f5564009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.337+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 msgr2=0x7f5578112990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.337+0000 7f557ce00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 0x7f5578112990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f556800c8c0 tx=0x7f55680044c0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 shutdown_connections 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f555c039480 0x7f555c03b930 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55780fbb10 0x7f5578112990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 --2- 172.21.15.44:0/4090473303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55780fcdc0 0x7f5578114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 >> 172.21.15.44:0/4090473303 conn(0x7f55780f7270 msgr2=0x7f55780f89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:26.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 shutdown_connections 2024-04-16T11:42:26.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.338+0000 7f557ce00700 1 -- 172.21.15.44:0/4090473303 wait complete. 2024-04-16T11:42:26.338 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: pgmap v7: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: Standby manager daemon smithi044.sczobv started 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/crt"}]: dispatch 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/key"}]: dispatch 2024-04-16T11:42:26.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:42:26.427 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3890512157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:42:26.427 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:26 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/4090473303' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 -- 172.21.15.44:0/1000632756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fce80 msgr2=0x7f0a940fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 --2- 172.21.15.44:0/1000632756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fce80 0x7f0a940fd290 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a84004940 tx=0x7f0a84009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 -- 172.21.15.44:0/1000632756 shutdown_connections 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 --2- 172.21.15.44:0/1000632756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fdff0 0x7f0a940fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 --2- 172.21.15.44:0/1000632756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fce80 0x7f0a940fd290 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 -- 172.21.15.44:0/1000632756 >> 172.21.15.44:0/1000632756 conn(0x7f0a940f8580 msgr2=0x7f0a940fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 -- 172.21.15.44:0/1000632756 shutdown_connections 2024-04-16T11:42:26.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.508+0000 7f0a9a000700 1 -- 172.21.15.44:0/1000632756 wait complete. 2024-04-16T11:42:26.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.509+0000 7f0a9a000700 1 Processor -- start 2024-04-16T11:42:26.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.509+0000 7f0a9a000700 1 -- start start 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.509+0000 7f0a9a000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fce80 0x7f0a94071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.509+0000 7f0a9a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 0x7f0a94070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.509+0000 7f0a9a000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94072260 con 0x7f0a940fdff0 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.509+0000 7f0a9a000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94070850 con 0x7f0a940fce80 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a92a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 0x7f0a94070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a92a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 0x7f0a94070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44730/0 (socket says 172.21.15.44:44730) 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a92a00700 1 -- 172.21.15.44:0/3617705766 learned_addr learned my addr 172.21.15.44:0/3617705766 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a93400700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fce80 0x7f0a94071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a92a00700 1 -- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fce80 msgr2=0x7f0a94071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a92a00700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fce80 0x7f0a94071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a92a00700 1 -- 172.21.15.44:0/3617705766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84005040 con 0x7f0a940fdff0 2024-04-16T11:42:26.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.510+0000 7f0a93400700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fce80 0x7f0a94071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:42:26.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.511+0000 7f0a92a00700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 0x7f0a94070310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000c830 tx=0x7f0a80004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:26.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.511+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8000a760 con 0x7f0a940fdff0 2024-04-16T11:42:26.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.511+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a94070b00 con 0x7f0a940fdff0 2024-04-16T11:42:26.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.511+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a940710c0 con 0x7f0a940fdff0 2024-04-16T11:42:26.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.511+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a8000ad90 con 0x7f0a940fdff0 2024-04-16T11:42:26.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.511+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a800113a0 con 0x7f0a940fdff0 2024-04-16T11:42:26.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.512+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a740052e0 con 0x7f0a940fdff0 2024-04-16T11:42:26.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.515+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7f0a8000a8c0 con 0x7f0a940fdff0 2024-04-16T11:42:26.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.515+0000 7f0a8b400700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0a7806bd00 0x7f0a7806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.516+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f0a8008a990 con 0x7f0a940fdff0 2024-04-16T11:42:26.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.516+0000 7f0a93400700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0a7806bd00 0x7f0a7806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.517+0000 7f0a93400700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0a7806bd00 0x7f0a7806e1b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400b7a0 tx=0x7f0a84009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:26.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.525+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f0a80057bc0 con 0x7f0a940fdff0 2024-04-16T11:42:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.778+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a74000bf0 con 0x7f0a7806bd00 2024-04-16T11:42:26.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.781+0000 7f0a8b400700 1 -- 172.21.15.44:0/3617705766 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f0a74000bf0 con 0x7f0a7806bd00 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "1/21 daemons upgraded", 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-16T11:42:26.780 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:42:26.781 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.784+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0a7806bd00 msgr2=0x7f0a7806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.784+0000 7f0a9a000700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0a7806bd00 0x7f0a7806e1b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400b7a0 tx=0x7f0a84009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.784+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 msgr2=0x7f0a94070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.784+0000 7f0a9a000700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 0x7f0a94070310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000c830 tx=0x7f0a80004470 comp rx=0 tx=0).stop 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 shutdown_connections 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0a7806bd00 0x7f0a7806e1b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a940fce80 0x7f0a94071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 --2- 172.21.15.44:0/3617705766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a940fdff0 0x7f0a94070310 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 >> 172.21.15.44:0/3617705766 conn(0x7f0a940f8580 msgr2=0x7f0a940f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 shutdown_connections 2024-04-16T11:42:26.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.785+0000 7f0a9a000700 1 -- 172.21.15.44:0/3617705766 wait complete. 2024-04-16T11:42:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: pgmap v7: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T11:42:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: Standby manager daemon smithi044.sczobv started 2024-04-16T11:42:26.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/crt"}]: dispatch 2024-04-16T11:42:26.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:42:26.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/key"}]: dispatch 2024-04-16T11:42:26.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/3524923395' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:42:26.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3890512157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:42:26.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:26 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/4090473303' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:42:26.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 -- 172.21.15.44:0/3124588878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fe0e0 msgr2=0x7ff6600fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 --2- 172.21.15.44:0/3124588878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fe0e0 0x7ff6600fe550 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff64c004940 tx=0x7ff64c009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:26.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 -- 172.21.15.44:0/3124588878 shutdown_connections 2024-04-16T11:42:26.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 --2- 172.21.15.44:0/3124588878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fe0e0 0x7ff6600fe550 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 --2- 172.21.15.44:0/3124588878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6600fcef0 0x7ff6600fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 -- 172.21.15.44:0/3124588878 >> 172.21.15.44:0/3124588878 conn(0x7ff6600f8530 msgr2=0x7ff6600fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:26.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.935+0000 7ff667c00700 1 -- 172.21.15.44:0/3124588878 shutdown_connections 2024-04-16T11:42:26.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.936+0000 7ff667c00700 1 -- 172.21.15.44:0/3124588878 wait complete. 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.936+0000 7ff667c00700 1 Processor -- start 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.936+0000 7ff667c00700 1 -- start start 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff667c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 0x7ff66010da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff667c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6600fe0e0 0x7ff66010dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff667c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66010e5d0 con 0x7ff6600fcef0 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff667c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66010edd0 con 0x7ff6600fe0e0 2024-04-16T11:42:26.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff665200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 0x7ff66010da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff665200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 0x7ff66010da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44748/0 (socket says 172.21.15.44:44748) 2024-04-16T11:42:26.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.937+0000 7ff665200700 1 -- 172.21.15.44:0/3694551999 learned_addr learned my addr 172.21.15.44:0/3694551999 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:26.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.938+0000 7ff65fe00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6600fe0e0 0x7ff66010dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.938+0000 7ff665200700 1 -- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6600fe0e0 msgr2=0x7ff66010dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:26.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.938+0000 7ff665200700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6600fe0e0 0x7ff66010dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:26.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.938+0000 7ff665200700 1 -- 172.21.15.44:0/3694551999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c005040 con 0x7ff6600fcef0 2024-04-16T11:42:26.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.939+0000 7ff665200700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 0x7ff66010da90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff65400c860 tx=0x7ff654004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:26.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.939+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff654011a80 con 0x7ff6600fcef0 2024-04-16T11:42:26.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.939+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff654011be0 con 0x7ff6600fcef0 2024-04-16T11:42:26.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.939+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66010f050 con 0x7ff6600fcef0 2024-04-16T11:42:26.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.939+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66010f610 con 0x7ff6600fcef0 2024-04-16T11:42:26.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.940+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff654019610 con 0x7ff6600fcef0 2024-04-16T11:42:26.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.941+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66005f5f0 con 0x7ff6600fcef0 2024-04-16T11:42:26.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.942+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7ff654017070 con 0x7ff6600fcef0 2024-04-16T11:42:26.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.943+0000 7ff65cc00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ff64806bcf0 0x7ff64806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:26.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.943+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7ff65408e830 con 0x7ff6600fcef0 2024-04-16T11:42:26.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.943+0000 7ff65fe00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ff64806bcf0 0x7ff64806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:26.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.944+0000 7ff65fe00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ff64806bcf0 0x7ff64806e1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00b3b0 tx=0x7ff64c009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:26.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:26.952+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7ff65405ba80 con 0x7ff6600fcef0 2024-04-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.277+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff6601059a0 con 0x7ff6600fcef0 2024-04-16T11:42:27.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.278+0000 7ff65cc00700 1 -- 172.21.15.44:0/3694551999 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff65405b610 con 0x7ff6600fcef0 2024-04-16T11:42:27.276 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:42:27.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ff64806bcf0 msgr2=0x7ff64806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:27.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ff64806bcf0 0x7ff64806e1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00b3b0 tx=0x7ff64c009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:27.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 msgr2=0x7ff66010da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:27.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 0x7ff66010da90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff65400c860 tx=0x7ff654004470 comp rx=0 tx=0).stop 2024-04-16T11:42:27.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 shutdown_connections 2024-04-16T11:42:27.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ff64806bcf0 0x7ff64806e1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:27.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6600fcef0 0x7ff66010da90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:27.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 --2- 172.21.15.44:0/3694551999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6600fe0e0 0x7ff66010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:27.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 >> 172.21.15.44:0/3694551999 conn(0x7ff6600f8530 msgr2=0x7ff6600f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:27.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 shutdown_connections 2024-04-16T11:42:27.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:27.281+0000 7ff667c00700 1 -- 172.21.15.44:0/3694551999 wait complete. 2024-04-16T11:42:27.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:27 smithi044 ceph-mon[28731]: mgrmap e24: smithi045.sdlwif(active, since 9s), standbys: smithi044.sczobv 2024-04-16T11:42:27.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:27 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:42:27.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:27 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3694551999' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:42:27.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:27 smithi045 ceph-mon[36308]: mgrmap e24: smithi045.sdlwif(active, since 9s), standbys: smithi044.sczobv 2024-04-16T11:42:27.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:27 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:42:27.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:27 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3694551999' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:42:28.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:28 smithi044 ceph-mon[28731]: pgmap v8: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T11:42:28.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:28 smithi044 ceph-mon[28731]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:28.801 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:28 smithi045 ceph-mon[36308]: pgmap v8: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T11:42:28.801 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:28 smithi045 ceph-mon[36308]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:30 smithi044 ceph-mon[28731]: pgmap v9: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T11:42:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:30 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:30 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:30.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:30 smithi045 ceph-mon[36308]: pgmap v9: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T11:42:30.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:30 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:30.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:30 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: pgmap v10: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:42:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi044 to 1559M 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:31 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: pgmap v10: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi044 to 1559M 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:31 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:33 smithi044 ceph-mon[28731]: pgmap v11: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:42:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:33 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:33 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:34.250 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:33 smithi045 ceph-mon[36308]: pgmap v11: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:42:34.250 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:33 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:34.250 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:33 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:35 smithi045 ceph-mon[36308]: pgmap v12: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:36.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:35 smithi044 ceph-mon[28731]: pgmap v12: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:36.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 68, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66741 2024-04-16T11:42:36.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 68, success, outsize: 120 2024-04-16T11:42:36.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 70, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66824 2024-04-16T11:42:36.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 70, success, outsize: 120 2024-04-16T11:42:36.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 72, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66910 2024-04-16T11:42:36.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 72, success, outsize: 120 2024-04-16T11:42:36.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 74, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67002 2024-04-16T11:42:36.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 74, success, outsize: 120 2024-04-16T11:42:36.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 76, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67091 2024-04-16T11:42:36.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 76, success, outsize: 120 2024-04-16T11:42:36.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 78, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67174 2024-04-16T11:42:36.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 78, success, outsize: 120 2024-04-16T11:42:37.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 80, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67260 2024-04-16T11:42:37.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 80, success, outsize: 120 2024-04-16T11:42:37.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 82, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67352 2024-04-16T11:42:37.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 82, success, outsize: 120 2024-04-16T11:42:37.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 84, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67441 2024-04-16T11:42:37.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 84, success, outsize: 120 2024-04-16T11:42:37.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 86, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67524 2024-04-16T11:42:37.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 86, success, outsize: 120 2024-04-16T11:42:37.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 88, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67610 2024-04-16T11:42:37.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 88, success, outsize: 120 2024-04-16T11:42:37.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 90, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67702 2024-04-16T11:42:37.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 90, success, outsize: 120 2024-04-16T11:42:37.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 92, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67791 2024-04-16T11:42:37.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 92, success, outsize: 120 2024-04-16T11:42:37.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67874 2024-04-16T11:42:37.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 94, success, outsize: 120 2024-04-16T11:42:37.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 96, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67960 2024-04-16T11:42:37.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 96, success, outsize: 120 2024-04-16T11:42:37.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 98, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68052 2024-04-16T11:42:37.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 98, success, outsize: 120 2024-04-16T11:42:37.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 100, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68141 2024-04-16T11:42:37.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 100, success, outsize: 120 2024-04-16T11:42:37.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 102, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68224 2024-04-16T11:42:37.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 102, success, outsize: 120 2024-04-16T11:42:37.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 104, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68310 2024-04-16T11:42:37.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 104, success, outsize: 120 2024-04-16T11:42:37.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 106, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68402 2024-04-16T11:42:37.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 106, success, outsize: 120 2024-04-16T11:42:37.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 108, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68491 2024-04-16T11:42:37.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 108, success, outsize: 120 2024-04-16T11:42:37.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 110, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68574 2024-04-16T11:42:37.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 110, success, outsize: 120 2024-04-16T11:42:37.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 112, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68660 2024-04-16T11:42:37.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 112, success, outsize: 120 2024-04-16T11:42:37.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 114, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68752 2024-04-16T11:42:37.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 114, success, outsize: 120 2024-04-16T11:42:37.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 116, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68841 2024-04-16T11:42:37.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 116, success, outsize: 120 2024-04-16T11:42:37.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 118, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68924 2024-04-16T11:42:37.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 118, success, outsize: 120 2024-04-16T11:42:37.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 120, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 69010 2024-04-16T11:42:37.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 120, success, outsize: 120 2024-04-16T11:42:37.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 122, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 69102 2024-04-16T11:42:37.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 122, success, outsize: 120 2024-04-16T11:42:38.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:37 smithi044 ceph-mon[28731]: pgmap v13: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:38.268 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:37 smithi045 ceph-mon[36308]: pgmap v13: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: Adjusting osd_memory_target on smithi045 to 2583M 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:42:39.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:39 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: Adjusting osd_memory_target on smithi045 to 2583M 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:42:39.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:42:39.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:42:39.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:39 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:42:40.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:40 smithi044 ceph-mon[28731]: pgmap v14: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:40 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:42:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:40 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:42:40.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:40 smithi045 ceph-mon[36308]: pgmap v14: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:40.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:40 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:42:40.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:40 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:42:41.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:41 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:42:41.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:41 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:42:41.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:41 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:42:41.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:41 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: pgmap v15: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.606 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.607 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.607 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:42 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: pgmap v15: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:42.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:42:42.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:42:42.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:42.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:42 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:43.720 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:43 smithi044 ceph-mon[28731]: Reconfiguring node-exporter.smithi044 (dependencies changed)... 2024-04-16T11:42:43.721 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:43 smithi044 ceph-mon[28731]: Reconfiguring daemon node-exporter.smithi044 on smithi044 2024-04-16T11:42:43.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:43 smithi045 ceph-mon[36308]: Reconfiguring node-exporter.smithi044 (dependencies changed)... 2024-04-16T11:42:43.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:43 smithi045 ceph-mon[36308]: Reconfiguring daemon node-exporter.smithi044 on smithi044 2024-04-16T11:42:44.644 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:44 smithi044 ceph-mon[28731]: pgmap v16: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:44.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:44 smithi045 ceph-mon[36308]: pgmap v16: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:46.676 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:46 smithi045 ceph-mon[36308]: pgmap v17: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:46.676 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:46 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:46.676 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:46 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:46.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:46 smithi044 ceph-mon[28731]: pgmap v17: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:46.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:46 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:46.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:46 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:47.686 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:47 smithi044 ceph-mon[28731]: Reconfiguring alertmanager.smithi044 (dependencies changed)... 2024-04-16T11:42:47.686 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:47 smithi044 ceph-mon[28731]: Reconfiguring daemon alertmanager.smithi044 on smithi044 2024-04-16T11:42:47.686 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:47 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:47.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:47 smithi045 ceph-mon[36308]: Reconfiguring alertmanager.smithi044 (dependencies changed)... 2024-04-16T11:42:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:47 smithi045 ceph-mon[36308]: Reconfiguring daemon alertmanager.smithi044 on smithi044 2024-04-16T11:42:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:47 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:42:48.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:48 smithi044 ceph-mon[28731]: pgmap v18: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:48.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:48 smithi045 ceph-mon[36308]: pgmap v18: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:50 smithi044 ceph-mon[28731]: pgmap v19: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:50.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:50 smithi045 ceph-mon[36308]: pgmap v19: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:52.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:52 smithi044 ceph-mon[28731]: pgmap v20: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:52.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:52 smithi045 ceph-mon[36308]: pgmap v20: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:53 smithi044 ceph-mon[28731]: pgmap v21: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:53 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:53 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:53 smithi044 ceph-mon[28731]: Reconfiguring grafana.smithi044 (dependencies changed)... 2024-04-16T11:42:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:53 smithi044 ceph-mon[28731]: Reconfiguring daemon grafana.smithi044 on smithi044 2024-04-16T11:42:54.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:53 smithi045 ceph-mon[36308]: pgmap v21: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:53 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:53 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:42:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:53 smithi045 ceph-mon[36308]: Reconfiguring grafana.smithi044 (dependencies changed)... 2024-04-16T11:42:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:53 smithi045 ceph-mon[36308]: Reconfiguring daemon grafana.smithi044 on smithi044 2024-04-16T11:42:55.952 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:55 smithi044 ceph-mon[28731]: pgmap v22: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:55 smithi045 ceph-mon[36308]: pgmap v22: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:57.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 -- 172.21.15.44:0/1157103100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c0fdb50 msgr2=0x7f8e6c0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:57.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 --2- 172.21.15.44:0/1157103100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c0fdb50 0x7f8e6c0fffd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8e58004940 tx=0x7f8e58009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:57.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 -- 172.21.15.44:0/1157103100 shutdown_connections 2024-04-16T11:42:57.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 --2- 172.21.15.44:0/1157103100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c0fdb50 0x7f8e6c0fffd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 --2- 172.21.15.44:0/1157103100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c0fd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 -- 172.21.15.44:0/1157103100 >> 172.21.15.44:0/1157103100 conn(0x7f8e6c0f7290 msgr2=0x7f8e6c0f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:57.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 -- 172.21.15.44:0/1157103100 shutdown_connections 2024-04-16T11:42:57.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.447+0000 7f8e72a00700 1 -- 172.21.15.44:0/1157103100 wait complete. 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.448+0000 7f8e72a00700 1 Processor -- start 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.448+0000 7f8e72a00700 1 -- start start 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.448+0000 7f8e72a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.448+0000 7f8e72a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e6c0fdb50 0x7f8e6c114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.448+0000 7f8e72a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1151d0 con 0x7f8e6c066ce0 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.448+0000 7f8e72a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c115340 con 0x7f8e6c0fdb50 2024-04-16T11:42:57.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e71600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e71600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36582/0 (socket says 172.21.15.44:36582) 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e71600700 1 -- 172.21.15.44:0/332301240 learned_addr learned my addr 172.21.15.44:0/332301240 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e70c00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e6c0fdb50 0x7f8e6c114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e71600700 1 -- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e6c0fdb50 msgr2=0x7f8e6c114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e71600700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e6c0fdb50 0x7f8e6c114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.449+0000 7f8e71600700 1 -- 172.21.15.44:0/332301240 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58005040 con 0x7f8e6c066ce0 2024-04-16T11:42:57.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.450+0000 7f8e71600700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c113f20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8e60002a50 tx=0x7f8e60004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:57.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.450+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e60007cc0 con 0x7f8e6c066ce0 2024-04-16T11:42:57.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.450+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e6000f040 con 0x7f8e6c066ce0 2024-04-16T11:42:57.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.450+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e6000daf0 con 0x7f8e6c066ce0 2024-04-16T11:42:57.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.450+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c1155c0 con 0x7f8e6c066ce0 2024-04-16T11:42:57.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.450+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c115b80 con 0x7f8e6c066ce0 2024-04-16T11:42:57.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.451+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c0faf30 con 0x7f8e6c066ce0 2024-04-16T11:42:57.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.452+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7f8e60009810 con 0x7f8e6c066ce0 2024-04-16T11:42:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.453+0000 7f8e69600700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f8e5406bd00 0x7f8e5406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.453+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f8e6008a800 con 0x7f8e6c066ce0 2024-04-16T11:42:57.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.454+0000 7f8e70c00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f8e5406bd00 0x7f8e5406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:57.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.455+0000 7f8e70c00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f8e5406bd00 0x7f8e5406e1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800b970 tx=0x7f8e58009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.458+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f8e60053d70 con 0x7f8e6c066ce0 2024-04-16T11:42:57.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.708+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e6c115df0 con 0x7f8e5406bd00 2024-04-16T11:42:57.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.712+0000 7f8e69600700 1 -- 172.21.15.44:0/332301240 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f8e6c115df0 con 0x7f8e5406bd00 2024-04-16T11:42:57.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f8e5406bd00 msgr2=0x7f8e5406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:57.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f8e5406bd00 0x7f8e5406e1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800b970 tx=0x7f8e58009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:57.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 msgr2=0x7f8e6c113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:57.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c113f20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8e60002a50 tx=0x7f8e60004600 comp rx=0 tx=0).stop 2024-04-16T11:42:57.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 shutdown_connections 2024-04-16T11:42:57.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f8e5406bd00 0x7f8e5406e1b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c066ce0 0x7f8e6c113f20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 --2- 172.21.15.44:0/332301240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e6c0fdb50 0x7f8e6c114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 >> 172.21.15.44:0/332301240 conn(0x7f8e6c0f7290 msgr2=0x7f8e6c0f76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 shutdown_connections 2024-04-16T11:42:57.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.715+0000 7f8e72a00700 1 -- 172.21.15.44:0/332301240 wait complete. 2024-04-16T11:42:57.731 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:42:57.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.886+0000 7fc145200700 1 -- 172.21.15.44:0/329803015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1401002a0 msgr2=0x7fc140100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.886+0000 7fc145200700 1 --2- 172.21.15.44:0/329803015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1401002a0 0x7fc140100710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc12c004940 tx=0x7fc12c009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.886+0000 7fc145200700 1 -- 172.21.15.44:0/329803015 shutdown_connections 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.887+0000 7fc145200700 1 --2- 172.21.15.44:0/329803015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1401002a0 0x7fc140100710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.887+0000 7fc145200700 1 --2- 172.21.15.44:0/329803015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1400feff0 0x7fc1400ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.887+0000 7fc145200700 1 -- 172.21.15.44:0/329803015 >> 172.21.15.44:0/329803015 conn(0x7fc1400fa790 msgr2=0x7fc1400fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.887+0000 7fc145200700 1 -- 172.21.15.44:0/329803015 shutdown_connections 2024-04-16T11:42:57.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.887+0000 7fc145200700 1 -- 172.21.15.44:0/329803015 wait complete. 2024-04-16T11:42:57.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc145200700 1 Processor -- start 2024-04-16T11:42:57.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc145200700 1 -- start start 2024-04-16T11:42:57.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc145200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 0x7fc14010fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:57.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc145200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1401002a0 0x7fc140110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:57.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc145200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc140110680 con 0x7fc1400feff0 2024-04-16T11:42:57.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc145200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1401107f0 con 0x7fc1401002a0 2024-04-16T11:42:57.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc13de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 0x7fc14010fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:57.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc13de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 0x7fc14010fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36600/0 (socket says 172.21.15.44:36600) 2024-04-16T11:42:57.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.888+0000 7fc13de00700 1 -- 172.21.15.44:0/1357084622 learned_addr learned my addr 172.21.15.44:0/1357084622 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc13d400700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1401002a0 0x7fc140110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc13de00700 1 -- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1401002a0 msgr2=0x7fc140110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc13de00700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1401002a0 0x7fc140110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc13de00700 1 -- 172.21.15.44:0/1357084622 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c005040 con 0x7fc1400feff0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc13d400700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1401002a0 0x7fc140110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc13de00700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 0x7fc14010fb40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc130002a50 tx=0x7fc1300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc130007cc0 con 0x7fc1400feff0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc13000f040 con 0x7fc1400feff0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.889+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13000db70 con 0x7fc1400feff0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.890+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc140115250 con 0x7fc1400feff0 2024-04-16T11:42:57.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.890+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc140115810 con 0x7fc1400feff0 2024-04-16T11:42:57.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.891+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14004c710 con 0x7fc1400feff0 2024-04-16T11:42:57.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.892+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7fc13000dd10 con 0x7fc1400feff0 2024-04-16T11:42:57.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.893+0000 7fc136000700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc12406bd00 0x7fc12406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:57.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.893+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fc13008a190 con 0x7fc1400feff0 2024-04-16T11:42:57.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.894+0000 7fc13d400700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc12406bd00 0x7fc12406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:57.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.895+0000 7fc13d400700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc12406bd00 0x7fc12406e1b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b7a0 tx=0x7fc12c009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:57.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:57.898+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fc130057230 con 0x7fc1400feff0 2024-04-16T11:42:57.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:57 smithi044 ceph-mon[28731]: pgmap v23: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:58.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.157+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1401045b0 con 0x7fc12406bd00 2024-04-16T11:42:58.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.161+0000 7fc136000700 1 -- 172.21.15.44:0/1357084622 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fc1401045b0 con 0x7fc12406bd00 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc12406bd00 msgr2=0x7fc12406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc12406bd00 0x7fc12406e1b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b7a0 tx=0x7fc12c009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 msgr2=0x7fc14010fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 0x7fc14010fb40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc130002a50 tx=0x7fc1300046b0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 shutdown_connections 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fc12406bd00 0x7fc12406e1b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1400feff0 0x7fc14010fb40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 --2- 172.21.15.44:0/1357084622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1401002a0 0x7fc140110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 >> 172.21.15.44:0/1357084622 conn(0x7fc1400fa790 msgr2=0x7fc1400fbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:58.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 shutdown_connections 2024-04-16T11:42:58.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.164+0000 7fc145200700 1 -- 172.21.15.44:0/1357084622 wait complete. 2024-04-16T11:42:58.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 -- 172.21.15.44:0/2805480632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fce80 msgr2=0x7f3ae40fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 --2- 172.21.15.44:0/2805480632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fce80 0x7f3ae40fd290 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004940 tx=0x7f3ad4009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:58.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 -- 172.21.15.44:0/2805480632 shutdown_connections 2024-04-16T11:42:58.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 --2- 172.21.15.44:0/2805480632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae40fdff0 0x7f3ae40fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 --2- 172.21.15.44:0/2805480632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fce80 0x7f3ae40fd290 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 -- 172.21.15.44:0/2805480632 >> 172.21.15.44:0/2805480632 conn(0x7f3ae40f8580 msgr2=0x7f3ae40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:58.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 -- 172.21.15.44:0/2805480632 shutdown_connections 2024-04-16T11:42:58.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.327+0000 7f3aeac00700 1 -- 172.21.15.44:0/2805480632 wait complete. 2024-04-16T11:42:58.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.329+0000 7f3aeac00700 1 Processor -- start 2024-04-16T11:42:58.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.329+0000 7f3aeac00700 1 -- start start 2024-04-16T11:42:58.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3aeac00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae40fce80 0x7f3ae410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:58.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3aeac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 0x7f3ae410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:58.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3aeac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae410e470 con 0x7f3ae40fdff0 2024-04-16T11:42:58.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3aeac00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae410e5e0 con 0x7f3ae40fce80 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3adbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 0x7f3ae410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3adbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 0x7f3ae410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36624/0 (socket says 172.21.15.44:36624) 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.330+0000 7f3adbe00700 1 -- 172.21.15.44:0/640448833 learned_addr learned my addr 172.21.15.44:0/640448833 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.331+0000 7f3ae3e00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae40fce80 0x7f3ae410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.331+0000 7f3adbe00700 1 -- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae40fce80 msgr2=0x7f3ae410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.331+0000 7f3adbe00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae40fce80 0x7f3ae410d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.331+0000 7f3adbe00700 1 -- 172.21.15.44:0/640448833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4005040 con 0x7f3ae40fdff0 2024-04-16T11:42:58.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.331+0000 7f3adbe00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 0x7f3ae410de70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00c830 tx=0x7f3acc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:57 smithi045 ceph-mon[36308]: pgmap v23: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:42:58.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.331+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc00a760 con 0x7f3ae40fdff0 2024-04-16T11:42:58.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.332+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3acc00ad90 con 0x7f3ae40fdff0 2024-04-16T11:42:58.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.332+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae4113040 con 0x7f3ae40fdff0 2024-04-16T11:42:58.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.332+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc0113a0 con 0x7f3ae40fdff0 2024-04-16T11:42:58.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.332+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae41135d0 con 0x7f3ae40fdff0 2024-04-16T11:42:58.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.333+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae404c710 con 0x7f3ae40fdff0 2024-04-16T11:42:58.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.334+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7f3acc012400 con 0x7f3ae40fdff0 2024-04-16T11:42:58.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.336+0000 7f3ae0c00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f3ac806bd00 0x7f3ac806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:58.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.336+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f3acc08a9e0 con 0x7f3ae40fdff0 2024-04-16T11:42:58.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.336+0000 7f3ae3e00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f3ac806bd00 0x7f3ac806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:58.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.337+0000 7f3ae3e00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f3ac806bd00 0x7f3ac806e1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004f90 tx=0x7f3ad4009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:58.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.339+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f3acc057c10 con 0x7f3ae40fdff0 2024-04-16T11:42:58.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.608+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ae4102400 con 0x7f3ac806bd00 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.638+0000 7f3ae0c00700 1 -- 172.21.15.44:0/640448833 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f3ae4102400 con 0x7f3ac806bd00 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 starting - - - - 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (10m) 29s ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (9m) 27s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (8m) 29s ago 10m 74.7M - 9.4.7 954c08fa6188 afde3dcbfd10 2024-04-16T11:42:58.636 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (3m) 29s ago 3m 11.8M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (3m) 29s ago 3m 22.3M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (3m) 27s ago 3m 11.8M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (3m) 27s ago 3m 11.9M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:9283 running (12m) 29s ago 12m 340M - 17.2.7-847-g45317f95 b333581ccdf5 53edec877549 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (53s) 27s ago 9m 488M - 19.0.0-2884-g218a6914 a4e4811815e4 de1d7adc1838 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (12m) 29s ago 12m 48.1M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (9m) 27s ago 9m 38.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 starting - - - - 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (9m) 27s ago 9m 7726k - 1.5.0 0da6a335fe13 e37d89867e75 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (8m) 29s ago 8m 45.8M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (7m) 29s ago 7m 48.3M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (6m) 29s ago 6m 45.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (6m) 27s ago 6m 44.9M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (5m) 27s ago 5m 42.7M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:42:58.637 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (5m) 27s ago 5m 44.7M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:42:58.638 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (8m) 29s ago 9m 49.9M - 2.43.0 a07b618ecd1d 99d239fdb386 2024-04-16T11:42:58.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f3ac806bd00 msgr2=0x7f3ac806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f3ac806bd00 0x7f3ac806e1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004f90 tx=0x7f3ad4009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:58.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 msgr2=0x7f3ae410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 0x7f3ae410de70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00c830 tx=0x7f3acc004470 comp rx=0 tx=0).stop 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 shutdown_connections 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f3ac806bd00 0x7f3ac806e1b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae40fce80 0x7f3ae410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 --2- 172.21.15.44:0/640448833 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ae40fdff0 0x7f3ae410de70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 >> 172.21.15.44:0/640448833 conn(0x7f3ae40f8580 msgr2=0x7f3ae40f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 shutdown_connections 2024-04-16T11:42:58.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.641+0000 7f3aeac00700 1 -- 172.21.15.44:0/640448833 wait complete. 2024-04-16T11:42:58.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 -- 172.21.15.44:0/1364020806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 msgr2=0x7fe208070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 --2- 172.21.15.44:0/1364020806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208070350 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8004940 tx=0x7fe1f8009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:58.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 -- 172.21.15.44:0/1364020806 shutdown_connections 2024-04-16T11:42:58.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 --2- 172.21.15.44:0/1364020806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 0x7fe208070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 --2- 172.21.15.44:0/1364020806 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208070350 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 -- 172.21.15.44:0/1364020806 >> 172.21.15.44:0/1364020806 conn(0x7fe2080f8540 msgr2=0x7fe2080fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:58.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 -- 172.21.15.44:0/1364020806 shutdown_connections 2024-04-16T11:42:58.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.805+0000 7fe20ee00700 1 -- 172.21.15.44:0/1364020806 wait complete. 2024-04-16T11:42:58.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.806+0000 7fe20ee00700 1 Processor -- start 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.806+0000 7fe20ee00700 1 -- start start 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.806+0000 7fe20ee00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 0x7fe20810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.806+0000 7fe20ee00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.806+0000 7fe20ee00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe208110860 con 0x7fe208071e50 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.806+0000 7fe20ee00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe208110fa0 con 0x7fe208070890 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36656/0 (socket says 172.21.15.44:36656) 2024-04-16T11:42:58.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207400700 1 -- 172.21.15.44:0/2286483782 learned_addr learned my addr 172.21.15.44:0/2286483782 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207e00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 0x7fe20810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207400700 1 -- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 msgr2=0x7fe20810fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207400700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 0x7fe20810fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207400700 1 -- 172.21.15.44:0/2286483782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8005040 con 0x7fe208071e50 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.807+0000 7fe207e00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 0x7fe20810fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.808+0000 7fe207400700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208110260 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4002a50 tx=0x7fe1f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.808+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1f4007cc0 con 0x7fe208071e50 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.808+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1f400f040 con 0x7fe208071e50 2024-04-16T11:42:58.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.808+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1f4009640 con 0x7fe208071e50 2024-04-16T11:42:58.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.808+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe208111280 con 0x7fe208071e50 2024-04-16T11:42:58.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.808+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe208111840 con 0x7fe208071e50 2024-04-16T11:42:58.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.809+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20805f5f0 con 0x7fe208071e50 2024-04-16T11:42:58.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.811+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7fe1f40097a0 con 0x7fe208071e50 2024-04-16T11:42:58.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.812+0000 7fe1ffe00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fe1ec06bcf0 0x7fe1ec06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:58.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.812+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fe1f408a850 con 0x7fe208071e50 2024-04-16T11:42:58.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.813+0000 7fe207e00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fe1ec06bcf0 0x7fe1ec06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:58.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.814+0000 7fe207e00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fe1ec06bcf0 0x7fe1ec06e1a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8004f90 tx=0x7fe1f8009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:58.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:58.817+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fe1f40578f0 con 0x7fe208071e50 2024-04-16T11:42:59.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.164+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe208111b60 con 0x7fe208071e50 2024-04-16T11:42:59.163 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:58 smithi044 ceph-mon[28731]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:59.163 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:58 smithi044 ceph-mon[28731]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.165+0000 7fe1ffe00700 1 -- 172.21.15.44:0/2286483782 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7fe1f4057480 con 0x7fe208071e50 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:59.164 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 13, 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:42:59.165 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.169+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fe1ec06bcf0 msgr2=0x7fe1ec06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.169+0000 7fe20ee00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fe1ec06bcf0 0x7fe1ec06e1a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8004f90 tx=0x7fe1f8009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 msgr2=0x7fe208110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208110260 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4002a50 tx=0x7fe1f40046b0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 shutdown_connections 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fe1ec06bcf0 0x7fe1ec06e1a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe208070890 0x7fe20810fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 --2- 172.21.15.44:0/2286483782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe208071e50 0x7fe208110260 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 >> 172.21.15.44:0/2286483782 conn(0x7fe2080f8540 msgr2=0x7fe2080f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:59.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 shutdown_connections 2024-04-16T11:42:59.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.170+0000 7fe20ee00700 1 -- 172.21.15.44:0/2286483782 wait complete. 2024-04-16T11:42:59.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:58 smithi045 ceph-mon[36308]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:59.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:58 smithi045 ceph-mon[36308]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:42:59.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 -- 172.21.15.44:0/1589632418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c070870 msgr2=0x7f572c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 --2- 172.21.15.44:0/1589632418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c070870 0x7f572c070ce0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5718004940 tx=0x7f5718009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:59.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 -- 172.21.15.44:0/1589632418 shutdown_connections 2024-04-16T11:42:59.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 --2- 172.21.15.44:0/1589632418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c070870 0x7f572c070ce0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 --2- 172.21.15.44:0/1589632418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f572c071e30 0x7f572c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 -- 172.21.15.44:0/1589632418 >> 172.21.15.44:0/1589632418 conn(0x7f572c0f8540 msgr2=0x7f572c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:59.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 -- 172.21.15.44:0/1589632418 shutdown_connections 2024-04-16T11:42:59.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.322+0000 7f5733000700 1 -- 172.21.15.44:0/1589632418 wait complete. 2024-04-16T11:42:59.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.323+0000 7f5733000700 1 Processor -- start 2024-04-16T11:42:59.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.323+0000 7f5733000700 1 -- start start 2024-04-16T11:42:59.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f5733000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f572c070870 0x7f572c10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:59.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f5733000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 0x7f572c110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:59.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f5733000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c110790 con 0x7f572c071e30 2024-04-16T11:42:59.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f5733000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c110fe0 con 0x7f572c070870 2024-04-16T11:42:59.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 0x7f572c110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:59.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 0x7f572c110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36682/0 (socket says 172.21.15.44:36682) 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572b400700 1 -- 172.21.15.44:0/1805878716 learned_addr learned my addr 172.21.15.44:0/1805878716 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572be00700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f572c070870 0x7f572c10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572b400700 1 -- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f572c070870 msgr2=0x7f572c10fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572b400700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f572c070870 0x7f572c10fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.324+0000 7f572b400700 1 -- 172.21.15.44:0/1805878716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718005040 con 0x7f572c071e30 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.325+0000 7f572b400700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 0x7f572c110190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5718009150 tx=0x7f5718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.325+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571800e040 con 0x7f572c071e30 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.325+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f571800bcc0 con 0x7f572c071e30 2024-04-16T11:42:59.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.325+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c111260 con 0x7f572c071e30 2024-04-16T11:42:59.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.325+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571801c630 con 0x7f572c071e30 2024-04-16T11:42:59.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.325+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c1117a0 con 0x7f572c071e30 2024-04-16T11:42:59.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.327+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c05f5f0 con 0x7f572c071e30 2024-04-16T11:42:59.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.328+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7f5718013420 con 0x7f572c071e30 2024-04-16T11:42:59.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.328+0000 7f5723e00700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f571006bcf0 0x7f571006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:59.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.329+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f571808b740 con 0x7f572c071e30 2024-04-16T11:42:59.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.329+0000 7f572be00700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f571006bcf0 0x7f571006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:59.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.330+0000 7f572be00700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f571006bcf0 0x7f571006e1a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f571c007fd0 tx=0x7f571c009040 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:59.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.333+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f57180589a0 con 0x7f572c071e30 2024-04-16T11:42:59.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.660+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f572c04bed0 con 0x7f572c071e30 2024-04-16T11:42:59.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.661+0000 7f5723e00700 1 -- 172.21.15.44:0/1805878716 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7f5718058530 con 0x7f572c071e30 2024-04-16T11:42:59.659 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:42:59.659 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:42:59.659 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:42:59.659 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:42:59.659 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:42:59.660 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:59.661 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:42:59.662 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:42:59.662 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:59.662 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:59.662 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.663+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f571006bcf0 msgr2=0x7f571006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.663+0000 7f5733000700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f571006bcf0 0x7f571006e1a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f571c007fd0 tx=0x7f571c009040 comp rx=0 tx=0).stop 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 msgr2=0x7f572c110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 0x7f572c110190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5718009150 tx=0x7f5718009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 shutdown_connections 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f571006bcf0 0x7f571006e1a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f572c070870 0x7f572c10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 --2- 172.21.15.44:0/1805878716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f572c071e30 0x7f572c110190 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 >> 172.21.15.44:0/1805878716 conn(0x7f572c0f8540 msgr2=0x7f572c0f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 shutdown_connections 2024-04-16T11:42:59.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.664+0000 7f5733000700 1 -- 172.21.15.44:0/1805878716 wait complete. 2024-04-16T11:42:59.664 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:42:59.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 -- 172.21.15.44:0/3882268713 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f4100320 msgr2=0x7f03f4100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 --2- 172.21.15.44:0/3882268713 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f4100320 0x7f03f4100790 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f03e0004940 tx=0x7f03e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:42:59.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 -- 172.21.15.44:0/3882268713 shutdown_connections 2024-04-16T11:42:59.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 --2- 172.21.15.44:0/3882268713 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f4100320 0x7f03f4100790 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 --2- 172.21.15.44:0/3882268713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f40ff070 0x7f03f40ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 -- 172.21.15.44:0/3882268713 >> 172.21.15.44:0/3882268713 conn(0x7f03f40fa950 msgr2=0x7f03f40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:42:59.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 -- 172.21.15.44:0/3882268713 shutdown_connections 2024-04-16T11:42:59.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.814+0000 7f03faa00700 1 -- 172.21.15.44:0/3882268713 wait complete. 2024-04-16T11:42:59.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.815+0000 7f03faa00700 1 Processor -- start 2024-04-16T11:42:59.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.815+0000 7f03faa00700 1 -- start start 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03faa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 0x7f03f410fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03faa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f4100320 0x7f03f4110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03faa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4110860 con 0x7f03f40ff070 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03faa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4110fa0 con 0x7f03f4100320 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 0x7f03f410fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03f3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 0x7f03f410fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36694/0 (socket says 172.21.15.44:36694) 2024-04-16T11:42:59.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03f3e00700 1 -- 172.21.15.44:0/3579421706 learned_addr learned my addr 172.21.15.44:0/3579421706 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:42:59.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.816+0000 7f03f3400700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f4100320 0x7f03f4110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:59.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03f3e00700 1 -- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f4100320 msgr2=0x7f03f4110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:42:59.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03f3e00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f4100320 0x7f03f4110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:42:59.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03f3e00700 1 -- 172.21.15.44:0/3579421706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e0005040 con 0x7f03f40ff070 2024-04-16T11:42:59.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03f3e00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 0x7f03f410fd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f03e4002a50 tx=0x7f03e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:59.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e4007cc0 con 0x7f03f40ff070 2024-04-16T11:42:59.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03e400f040 con 0x7f03f40ff070 2024-04-16T11:42:59.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f4111280 con 0x7f03f40ff070 2024-04-16T11:42:59.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e400da70 con 0x7f03f40ff070 2024-04-16T11:42:59.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.817+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f4111840 con 0x7f03f40ff070 2024-04-16T11:42:59.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.819+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f405f5f0 con 0x7f03f40ff070 2024-04-16T11:42:59.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.821+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7f03e400dc10 con 0x7f03f40ff070 2024-04-16T11:42:59.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.822+0000 7f03ebe00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f03d806bdc0 0x7f03d806e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:42:59.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.822+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f03e408a120 con 0x7f03f40ff070 2024-04-16T11:42:59.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.823+0000 7f03f3400700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f03d806bdc0 0x7f03d806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:42:59.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.824+0000 7f03f3400700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f03d806bdc0 0x7f03d806e270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f03e000b3b0 tx=0x7f03e0009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:42:59.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:42:59.827+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f03e404f930 con 0x7f03f40ff070 2024-04-16T11:43:00.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.074+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03f4111bc0 con 0x7f03d806bdc0 2024-04-16T11:43:00.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:59 smithi044 ceph-mon[28731]: pgmap v24: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:00.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:59 smithi044 ceph-mon[28731]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:00.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:59 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2286483782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:43:00.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:59 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:00.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:59 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:00.073 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:42:59 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/1805878716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.080+0000 7f03ebe00700 1 -- 172.21.15.44:0/3579421706 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f03f4111bc0 con 0x7f03d806bdc0 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "1/21 daemons upgraded", 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:43:00.078 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.082+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f03d806bdc0 msgr2=0x7f03d806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.082+0000 7f03faa00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f03d806bdc0 0x7f03d806e270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f03e000b3b0 tx=0x7f03e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 msgr2=0x7f03f410fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 0x7f03f410fd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f03e4002a50 tx=0x7f03e40046b0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 shutdown_connections 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f03d806bdc0 0x7f03d806e270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03f40ff070 0x7f03f410fd20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 --2- 172.21.15.44:0/3579421706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03f4100320 0x7f03f4110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 >> 172.21.15.44:0/3579421706 conn(0x7f03f40fa950 msgr2=0x7f03f40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:00.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 shutdown_connections 2024-04-16T11:43:00.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.083+0000 7f03faa00700 1 -- 172.21.15.44:0/3579421706 wait complete. 2024-04-16T11:43:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.234+0000 7f674ac00700 1 -- 172.21.15.44:0/2075950486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 msgr2=0x7f67440fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.234+0000 7f674ac00700 1 --2- 172.21.15.44:0/2075950486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f67440fe550 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6730004940 tx=0x7f6730009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.234+0000 7f674ac00700 1 -- 172.21.15.44:0/2075950486 shutdown_connections 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.234+0000 7f674ac00700 1 --2- 172.21.15.44:0/2075950486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f67440fe550 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.234+0000 7f674ac00700 1 --2- 172.21.15.44:0/2075950486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 0x7f67440fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.234+0000 7f674ac00700 1 -- 172.21.15.44:0/2075950486 >> 172.21.15.44:0/2075950486 conn(0x7f67440f8530 msgr2=0x7f67440fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.235+0000 7f674ac00700 1 -- 172.21.15.44:0/2075950486 shutdown_connections 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.235+0000 7f674ac00700 1 -- 172.21.15.44:0/2075950486 wait complete. 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.236+0000 7f674ac00700 1 Processor -- start 2024-04-16T11:43:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.236+0000 7f674ac00700 1 -- start start 2024-04-16T11:43:00.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.236+0000 7f674ac00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 0x7f674410da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:00.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.236+0000 7f674ac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f674410dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:00.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.236+0000 7f674ac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674410e5d0 con 0x7f67440fe0e0 2024-04-16T11:43:00.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.236+0000 7f674ac00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674410edd0 con 0x7f67440fcef0 2024-04-16T11:43:00.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f674410dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:00.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f674410dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36702/0 (socket says 172.21.15.44:36702) 2024-04-16T11:43:00.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743400700 1 -- 172.21.15.44:0/2746443891 learned_addr learned my addr 172.21.15.44:0/2746443891 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:00.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743e00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 0x7f674410da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743400700 1 -- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 msgr2=0x7f674410da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743400700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 0x7f674410da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743400700 1 -- 172.21.15.44:0/2746443891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730005040 con 0x7f67440fe0e0 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.237+0000 7f6743e00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 0x7f674410da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.238+0000 7f6743400700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f674410dfd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f673000b3b0 tx=0x7f6730009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.238+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673000e040 con 0x7f67440fe0e0 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.238+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674410f050 con 0x7f67440fe0e0 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.238+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674410f590 con 0x7f67440fe0e0 2024-04-16T11:43:00.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.238+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f673000bcc0 con 0x7f67440fe0e0 2024-04-16T11:43:00.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.238+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673001c630 con 0x7f67440fe0e0 2024-04-16T11:43:00.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.240+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67240052e0 con 0x7f67440fe0e0 2024-04-16T11:43:00.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.241+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) v1 ==== 89650+0+0 (secure 0 0 0) 0x7f6730026440 con 0x7f67440fe0e0 2024-04-16T11:43:00.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.242+0000 7f673be00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f672806bcf0 0x7f672806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:00.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.242+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f673008b5a0 con 0x7f67440fe0e0 2024-04-16T11:43:00.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.243+0000 7f6743e00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f672806bcf0 0x7f672806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:00.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.244+0000 7f6743e00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f672806bcf0 0x7f672806e1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f67440fcc30 tx=0x7f673400b040 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:00.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.252+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f6730058800 con 0x7f67440fe0e0 2024-04-16T11:43:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:59 smithi045 ceph-mon[36308]: pgmap v24: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:59 smithi045 ceph-mon[36308]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:59 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2286483782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:43:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:59 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:59 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:42:59 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/1805878716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:43:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.575+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6724005860 con 0x7f67440fe0e0 2024-04-16T11:43:00.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.575+0000 7f673be00700 1 -- 172.21.15.44:0/2746443891 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6730058390 con 0x7f67440fe0e0 2024-04-16T11:43:00.573 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f672806bcf0 msgr2=0x7f672806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f672806bcf0 0x7f672806e1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f67440fcc30 tx=0x7f673400b040 comp rx=0 tx=0).stop 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 msgr2=0x7f674410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f674410dfd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f673000b3b0 tx=0x7f6730009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 shutdown_connections 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f672806bcf0 0x7f672806e1a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67440fcef0 0x7f674410da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 --2- 172.21.15.44:0/2746443891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f67440fe0e0 0x7f674410dfd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:00.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 >> 172.21.15.44:0/2746443891 conn(0x7f67440f8530 msgr2=0x7f67440f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:00.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 shutdown_connections 2024-04-16T11:43:00.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:00.579+0000 7f674ac00700 1 -- 172.21.15.44:0/2746443891 wait complete. 2024-04-16T11:43:01.134 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:00 smithi044 ceph-mon[28731]: Reconfiguring prometheus.smithi044 (dependencies changed)... 2024-04-16T11:43:01.134 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:00 smithi044 ceph-mon[28731]: Reconfiguring daemon prometheus.smithi044 on smithi044 2024-04-16T11:43:01.134 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:00 smithi044 ceph-mon[28731]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:01.134 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:00 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2746443891' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:43:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:00 smithi045 ceph-mon[36308]: Reconfiguring prometheus.smithi044 (dependencies changed)... 2024-04-16T11:43:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:00 smithi045 ceph-mon[36308]: Reconfiguring daemon prometheus.smithi044 on smithi044 2024-04-16T11:43:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:00 smithi045 ceph-mon[36308]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:00 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2746443891' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:43:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:01 smithi044 ceph-mon[28731]: pgmap v25: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:01 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:01 smithi045 ceph-mon[36308]: pgmap v25: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:01 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:04.200 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:03 smithi044 ceph-mon[28731]: pgmap v26: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:03 smithi045 ceph-mon[36308]: pgmap v26: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:06.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:05 smithi044 ceph-mon[28731]: pgmap v27: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:05 smithi045 ceph-mon[36308]: pgmap v27: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:07 smithi044 ceph-mon[28731]: pgmap v28: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:07 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:07 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:07 smithi044 ceph-mon[28731]: Reconfiguring node-exporter.smithi045 (dependencies changed)... 2024-04-16T11:43:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:07 smithi044 ceph-mon[28731]: Reconfiguring daemon node-exporter.smithi045 on smithi045 2024-04-16T11:43:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:07 smithi045 ceph-mon[36308]: pgmap v28: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:08.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:07 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:08.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:07 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:08.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:07 smithi045 ceph-mon[36308]: Reconfiguring node-exporter.smithi045 (dependencies changed)... 2024-04-16T11:43:08.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:07 smithi045 ceph-mon[36308]: Reconfiguring daemon node-exporter.smithi045 on smithi045 2024-04-16T11:43:10.045 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:09 smithi045 ceph-mon[36308]: pgmap v29: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:10.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:09 smithi044 ceph-mon[28731]: pgmap v29: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:11.819 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: pgmap v30: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: Upgrade: Need to upgrade myself (mgr.smithi045.sdlwif) 2024-04-16T11:43:11.820 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:11 smithi044 ceph-mon[28731]: Upgrade: Need to upgrade myself (mgr.smithi045.sdlwif) 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: pgmap v30: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: Upgrade: Need to upgrade myself (mgr.smithi045.sdlwif) 2024-04-16T11:43:11.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:11 smithi045 ceph-mon[36308]: Upgrade: Need to upgrade myself (mgr.smithi045.sdlwif) 2024-04-16T11:43:14.109 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: pgmap v31: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:14.109 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: Upgrade: Updating mgr.smithi044.sczobv 2024-04-16T11:43:14.109 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:14.109 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:43:14.110 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:43:14.110 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:43:14.110 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:43:14.110 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:14 smithi044 ceph-mon[28731]: Deploying daemon mgr.smithi044.sczobv on smithi044 2024-04-16T11:43:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: pgmap v31: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: Upgrade: Updating mgr.smithi044.sczobv 2024-04-16T11:43:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:43:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:43:14.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:43:14.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:43:14.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:14 smithi045 ceph-mon[36308]: Deploying daemon mgr.smithi044.sczobv on smithi044 2024-04-16T11:43:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:16 smithi045 ceph-mon[36308]: pgmap v32: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:16.386 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:16 smithi044 ceph-mon[28731]: pgmap v32: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:17.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:17 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:17.448 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:17 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:18.217 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:18 smithi044 ceph-mon[28731]: pgmap v33: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:18.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:18 smithi045 ceph-mon[36308]: pgmap v33: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:20.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:19 smithi044 ceph-mon[28731]: pgmap v34: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:20.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:19 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:19 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:19 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:20.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:19 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:19 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:19 smithi045 ceph-mon[36308]: pgmap v34: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:20.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:19 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:19 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:19 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:20.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:19 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:20.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:19 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:22.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:22 smithi044 ceph-mon[28731]: pgmap v35: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:22 smithi045 ceph-mon[36308]: pgmap v35: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:24.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:24 smithi044 ceph-mon[28731]: pgmap v36: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:24.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:24 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:24.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:24 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:24.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:24 smithi045 ceph-mon[36308]: pgmap v36: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:24 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:24 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:26.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:26 smithi044 ceph-mon[28731]: pgmap v37: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:26.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:26 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:26.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:26 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:26.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:26 smithi045 ceph-mon[36308]: pgmap v37: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:26 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:26.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:26 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:28.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:28 smithi044 ceph-mon[28731]: pgmap v38: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:28.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:28 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:28.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:28 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:28.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:28 smithi045 ceph-mon[36308]: pgmap v38: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:28.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:28 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:28.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:28 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:29 smithi044 ceph-mon[28731]: Standby manager daemon smithi044.sczobv restarted 2024-04-16T11:43:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:29 smithi044 ceph-mon[28731]: Standby manager daemon smithi044.sczobv started 2024-04-16T11:43:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:29 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/crt"}]: dispatch 2024-04-16T11:43:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:29 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:43:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:29 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/key"}]: dispatch 2024-04-16T11:43:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:29 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:43:29.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:29 smithi045 ceph-mon[36308]: Standby manager daemon smithi044.sczobv restarted 2024-04-16T11:43:29.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:29 smithi045 ceph-mon[36308]: Standby manager daemon smithi044.sczobv started 2024-04-16T11:43:29.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:29 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/crt"}]: dispatch 2024-04-16T11:43:29.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:29 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:43:29.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:29 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi044.sczobv/key"}]: dispatch 2024-04-16T11:43:29.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:29 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:43:30.767 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:30 smithi044 ceph-mon[28731]: pgmap v39: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:30.767 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:30 smithi044 ceph-mon[28731]: mgrmap e25: smithi045.sdlwif(active, since 72s), standbys: smithi044.sczobv 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 -- 172.21.15.44:0/2514754617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa00709a0 msgr2=0x7ffaa0070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 --2- 172.21.15.44:0/2514754617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa00709a0 0x7ffaa0070e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c004940 tx=0x7ffa8c009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 -- 172.21.15.44:0/2514754617 shutdown_connections 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 --2- 172.21.15.44:0/2514754617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa00709a0 0x7ffaa0070e10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 --2- 172.21.15.44:0/2514754617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa0071f50 0x7ffaa0070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 -- 172.21.15.44:0/2514754617 >> 172.21.15.44:0/2514754617 conn(0x7ffaa00f8530 msgr2=0x7ffaa00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 -- 172.21.15.44:0/2514754617 shutdown_connections 2024-04-16T11:43:30.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.769+0000 7ffaa7000700 1 -- 172.21.15.44:0/2514754617 wait complete. 2024-04-16T11:43:30.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.771+0000 7ffaa7000700 1 Processor -- start 2024-04-16T11:43:30.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.771+0000 7ffaa7000700 1 -- start start 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.771+0000 7ffaa7000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00709a0 0x7ffaa0114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.771+0000 7ffaa7000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 0x7ffaa0114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.771+0000 7ffaa7000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa0114b80 con 0x7ffaa0071f50 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.771+0000 7ffaa7000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa01153d0 con 0x7ffaa00709a0 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 0x7ffaa0114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 0x7ffaa0114580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54738/0 (socket says 172.21.15.44:54738) 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9f400700 1 -- 172.21.15.44:0/446994578 learned_addr learned my addr 172.21.15.44:0/446994578 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:30.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9fe00700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00709a0 0x7ffaa0114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9f400700 1 -- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00709a0 msgr2=0x7ffaa0114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9f400700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00709a0 0x7ffaa0114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.772+0000 7ffa9f400700 1 -- 172.21.15.44:0/446994578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c005040 con 0x7ffaa0071f50 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.773+0000 7ffa9f400700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 0x7ffaa0114580 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c00b3b0 tx=0x7ffa8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.773+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa8c00e040 con 0x7ffaa0071f50 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.773+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa8c00bbf0 con 0x7ffaa0071f50 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.773+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa8c01c630 con 0x7ffaa0071f50 2024-04-16T11:43:30.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.773+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa0115650 con 0x7ffaa0071f50 2024-04-16T11:43:30.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.774+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa0115b40 con 0x7ffaa0071f50 2024-04-16T11:43:30.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.775+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa005f5f0 con 0x7ffaa0071f50 2024-04-16T11:43:30.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.776+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7ffa8c00bd60 con 0x7ffaa0071f50 2024-04-16T11:43:30.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.777+0000 7ffa97e00700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ffa8406e620 0x7ffa84070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:30.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.777+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7ffa8c08ec60 con 0x7ffaa0071f50 2024-04-16T11:43:30.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.778+0000 7ffa9fe00700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ffa8406e620 0x7ffa84070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:30.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.779+0000 7ffa9fe00700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ffa8406e620 0x7ffa84070ad0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0071c90 tx=0x7ffa9000b040 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:30.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:30.783+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7ffa8c05bd00 con 0x7ffaa0071f50 2024-04-16T11:43:30.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:30 smithi045 ceph-mon[36308]: pgmap v39: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:30.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:30 smithi045 ceph-mon[36308]: mgrmap e25: smithi045.sdlwif(active, since 72s), standbys: smithi044.sczobv 2024-04-16T11:43:31.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.039+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaa0115ee0 con 0x7ffa8406e620 2024-04-16T11:43:31.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.042+0000 7ffa97e00700 1 -- 172.21.15.44:0/446994578 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7ffaa0115ee0 con 0x7ffa8406e620 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ffa8406e620 msgr2=0x7ffa84070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ffa8406e620 0x7ffa84070ad0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0071c90 tx=0x7ffa9000b040 comp rx=0 tx=0).stop 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 msgr2=0x7ffaa0114580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 0x7ffaa0114580 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c00b3b0 tx=0x7ffa8c009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 shutdown_connections 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7ffa8406e620 0x7ffa84070ad0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00709a0 0x7ffaa0114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 --2- 172.21.15.44:0/446994578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0071f50 0x7ffaa0114580 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.044+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 >> 172.21.15.44:0/446994578 conn(0x7ffaa00f8530 msgr2=0x7ffaa00f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.045+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 shutdown_connections 2024-04-16T11:43:31.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.045+0000 7ffaa7000700 1 -- 172.21.15.44:0/446994578 wait complete. 2024-04-16T11:43:31.058 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:43:31.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.208+0000 7fce16200700 1 -- 172.21.15.44:0/1620936224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce100ff020 msgr2=0x7fce100ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.208+0000 7fce16200700 1 --2- 172.21.15.44:0/1620936224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce100ff020 0x7fce100ff430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fce00004940 tx=0x7fce00009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.209+0000 7fce16200700 1 -- 172.21.15.44:0/1620936224 shutdown_connections 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.209+0000 7fce16200700 1 --2- 172.21.15.44:0/1620936224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce101002d0 0x7fce10100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.209+0000 7fce16200700 1 --2- 172.21.15.44:0/1620936224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce100ff020 0x7fce100ff430 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.209+0000 7fce16200700 1 -- 172.21.15.44:0/1620936224 >> 172.21.15.44:0/1620936224 conn(0x7fce100fa950 msgr2=0x7fce100fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.209+0000 7fce16200700 1 -- 172.21.15.44:0/1620936224 shutdown_connections 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.209+0000 7fce16200700 1 -- 172.21.15.44:0/1620936224 wait complete. 2024-04-16T11:43:31.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.210+0000 7fce16200700 1 Processor -- start 2024-04-16T11:43:31.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.210+0000 7fce16200700 1 -- start start 2024-04-16T11:43:31.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.210+0000 7fce16200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce100ff020 0x7fce1010fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.210+0000 7fce16200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 0x7fce10110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.210+0000 7fce16200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10110620 con 0x7fce101002d0 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.210+0000 7fce16200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10110790 con 0x7fce100ff020 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 0x7fce10110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 0x7fce10110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54756/0 (socket says 172.21.15.44:54756) 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 -- 172.21.15.44:0/104684543 learned_addr learned my addr 172.21.15.44:0/104684543 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:31.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0f400700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce100ff020 0x7fce1010fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 -- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce100ff020 msgr2=0x7fce1010fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce100ff020 0x7fce1010fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 -- 172.21.15.44:0/104684543 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc005040 con 0x7fce101002d0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0f400700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce100ff020 0x7fce1010fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.211+0000 7fce0ea00700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 0x7fce10110020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0096d0 tx=0x7fcdfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.212+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdfc00e840 con 0x7fce101002d0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.212+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdfc00f040 con 0x7fce101002d0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.212+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdfc013e50 con 0x7fce101002d0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.212+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce00005040 con 0x7fce101002d0 2024-04-16T11:43:31.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.212+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce10115620 con 0x7fce101002d0 2024-04-16T11:43:31.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.213+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1004c710 con 0x7fce101002d0 2024-04-16T11:43:31.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.214+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fcdfc00e9a0 con 0x7fce101002d0 2024-04-16T11:43:31.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.215+0000 7fce07400700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fcdf406e630 0x7fcdf4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:31.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.215+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fcdfc08de30 con 0x7fce101002d0 2024-04-16T11:43:31.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.216+0000 7fce0f400700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fcdf406e630 0x7fcdf4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:31.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.217+0000 7fce0f400700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fcdf406e630 0x7fcdf4070ae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fce00009fd0 tx=0x7fce00009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:31.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.220+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fcdfc05aed0 con 0x7fce101002d0 2024-04-16T11:43:31.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.479+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce101045a0 con 0x7fcdf406e630 2024-04-16T11:43:31.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.483+0000 7fce07400700 1 -- 172.21.15.44:0/104684543 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fce101045a0 con 0x7fcdf406e630 2024-04-16T11:43:31.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fcdf406e630 msgr2=0x7fcdf4070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fcdf406e630 0x7fcdf4070ae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fce00009fd0 tx=0x7fce00009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 msgr2=0x7fce10110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 0x7fce10110020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0096d0 tx=0x7fcdfc009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 shutdown_connections 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fcdf406e630 0x7fcdf4070ae0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce100ff020 0x7fce1010fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 --2- 172.21.15.44:0/104684543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce101002d0 0x7fce10110020 secure :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0096d0 tx=0x7fcdfc009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.486+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 >> 172.21.15.44:0/104684543 conn(0x7fce100fa950 msgr2=0x7fce100fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:31.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.487+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 shutdown_connections 2024-04-16T11:43:31.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.487+0000 7fce16200700 1 -- 172.21.15.44:0/104684543 wait complete. 2024-04-16T11:43:31.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 -- 172.21.15.44:0/4125021649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 msgr2=0x7f3464070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 --2- 172.21.15.44:0/4125021649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464070e10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3450004940 tx=0x7f3450009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 -- 172.21.15.44:0/4125021649 shutdown_connections 2024-04-16T11:43:31.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 --2- 172.21.15.44:0/4125021649 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464070e10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 --2- 172.21.15.44:0/4125021649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3464071f50 0x7f3464070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 -- 172.21.15.44:0/4125021649 >> 172.21.15.44:0/4125021649 conn(0x7f34640f8530 msgr2=0x7f34640fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 -- 172.21.15.44:0/4125021649 shutdown_connections 2024-04-16T11:43:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.633+0000 7f3469200700 1 -- 172.21.15.44:0/4125021649 wait complete. 2024-04-16T11:43:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.634+0000 7f3469200700 1 Processor -- start 2024-04-16T11:43:31.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.634+0000 7f3469200700 1 -- start start 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3469200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3469200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3464071f50 0x7f3464114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3469200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464114b80 con 0x7f34640709a0 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3469200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34641153d0 con 0x7f3464071f50 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3461e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3461e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41238/0 (socket says 172.21.15.44:41238) 2024-04-16T11:43:31.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.635+0000 7f3461e00700 1 -- 172.21.15.44:0/13995900 learned_addr learned my addr 172.21.15.44:0/13995900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.636+0000 7f3461400700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3464071f50 0x7f3464114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.636+0000 7f3461e00700 1 -- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3464071f50 msgr2=0x7f3464114580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.636+0000 7f3461e00700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3464071f50 0x7f3464114580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.636+0000 7f3461e00700 1 -- 172.21.15.44:0/13995900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3450005040 con 0x7f34640709a0 2024-04-16T11:43:31.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.636+0000 7f3461e00700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464114040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f345400c8b0 tx=0x7f3454004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:31.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.637+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f345400a760 con 0x7f34640709a0 2024-04-16T11:43:31.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.637+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f345400ad90 con 0x7f34640709a0 2024-04-16T11:43:31.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.637+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34540113a0 con 0x7f34640709a0 2024-04-16T11:43:31.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.637+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34641156b0 con 0x7f34640709a0 2024-04-16T11:43:31.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.637+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3464115bf0 con 0x7f34640709a0 2024-04-16T11:43:31.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.638+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346405f5f0 con 0x7f34640709a0 2024-04-16T11:43:31.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.639+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f3454012400 con 0x7f34640709a0 2024-04-16T11:43:31.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.640+0000 7f345a000700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f344806e620 0x7f3448070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:31.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.640+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f345408de70 con 0x7f34640709a0 2024-04-16T11:43:31.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.641+0000 7f3461400700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f344806e620 0x7f3448070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:31.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.642+0000 7f3461400700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f344806e620 0x7f3448070ad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f345000b3b0 tx=0x7f3450009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:31.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.645+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f345405b0a0 con 0x7f34640709a0 2024-04-16T11:43:31.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.885+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3464116090 con 0x7f344806e620 2024-04-16T11:43:31.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.915+0000 7f345a000700 1 -- 172.21.15.44:0/13995900 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3464116090 con 0x7f344806e620 2024-04-16T11:43:31.913 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:43:31.913 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (39s) 5s ago 11m 14.0M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:43:31.913 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (11m) 5s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (9m) 7s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (32s) 5s ago 10m 71.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (3m) 5s ago 3m 12.1M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (4m) 5s ago 3m 22.7M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (3m) 7s ago 3m 12.1M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (3m) 7s ago 3m 12.2M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (13s) 5s ago 12m 69.3M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (86s) 7s ago 9m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 de1d7adc1838 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (13m) 5s ago 13m 48.9M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (9m) 7s ago 9m 41.3M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (46s) 5s ago 11m 7444k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (21s) 7s ago 9m 6744k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (8m) 5s ago 8m 46.1M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (8m) 5s ago 8m 48.3M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (7m) 5s ago 7m 45.3M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:43:31.914 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (6m) 7s ago 6m 45.2M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:43:31.915 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (6m) 7s ago 6m 43.0M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:43:31.915 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (5m) 7s ago 5m 45.1M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:43:31.915 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (25s) 5s ago 10m 41.3M - 2.43.0 a07b618ecd1d 83145f1debe8 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f344806e620 msgr2=0x7f3448070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f344806e620 0x7f3448070ad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f345000b3b0 tx=0x7f3450009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 msgr2=0x7f3464114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464114040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f345400c8b0 tx=0x7f3454004470 comp rx=0 tx=0).stop 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 shutdown_connections 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f344806e620 0x7f3448070ad0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34640709a0 0x7f3464114040 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 --2- 172.21.15.44:0/13995900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3464071f50 0x7f3464114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:31.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 >> 172.21.15.44:0/13995900 conn(0x7f34640f8530 msgr2=0x7f34640f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:31.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 shutdown_connections 2024-04-16T11:43:31.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:31.918+0000 7f3469200700 1 -- 172.21.15.44:0/13995900 wait complete. 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 -- 172.21.15.44:0/3913175847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a040fffb0 msgr2=0x7f6a04102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3913175847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a040fffb0 0x7f6a04102390 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f69f0004940 tx=0x7f69f0009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 -- 172.21.15.44:0/3913175847 shutdown_connections 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3913175847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a040fffb0 0x7f6a04102390 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3913175847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a04066ee0 0x7f6a040ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 -- 172.21.15.44:0/3913175847 >> 172.21.15.44:0/3913175847 conn(0x7f6a040f7480 msgr2=0x7f6a040f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:32.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 -- 172.21.15.44:0/3913175847 shutdown_connections 2024-04-16T11:43:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.066+0000 7f6a09c00700 1 -- 172.21.15.44:0/3913175847 wait complete. 2024-04-16T11:43:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.067+0000 7f6a09c00700 1 Processor -- start 2024-04-16T11:43:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.067+0000 7f6a09c00700 1 -- start start 2024-04-16T11:43:32.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a09c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 0x7f6a0410fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:32.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a09c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040fffb0 0x7f6a04110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:32.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a09c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04110750 con 0x7f6a04066ee0 2024-04-16T11:43:32.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a09c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a041108c0 con 0x7f6a040fffb0 2024-04-16T11:43:32.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a02a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 0x7f6a0410fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a02a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 0x7f6a0410fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41262/0 (socket says 172.21.15.44:41262) 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a02a00700 1 -- 172.21.15.44:0/3555880223 learned_addr learned my addr 172.21.15.44:0/3555880223 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a02000700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040fffb0 0x7f6a04110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a02a00700 1 -- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040fffb0 msgr2=0x7f6a04110150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.068+0000 7f6a02a00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040fffb0 0x7f6a04110150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f6a02a00700 1 -- 172.21.15.44:0/3555880223 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0005040 con 0x7f6a04066ee0 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f6a02a00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 0x7f6a0410fc10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f69f400c800 tx=0x7f69f4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69f400a760 con 0x7f6a04066ee0 2024-04-16T11:43:32.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a041152d0 con 0x7f6a04066ee0 2024-04-16T11:43:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69f400ad90 con 0x7f6a04066ee0 2024-04-16T11:43:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69f40113a0 con 0x7f6a04066ee0 2024-04-16T11:43:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.069+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04115890 con 0x7f6a04066ee0 2024-04-16T11:43:32.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.071+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0404c710 con 0x7f6a04066ee0 2024-04-16T11:43:32.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.072+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f69f4012400 con 0x7f6a04066ee0 2024-04-16T11:43:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.073+0000 7f69faa00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f69e806e630 0x7f69e8070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:32.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.073+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f69f408de20 con 0x7f6a04066ee0 2024-04-16T11:43:32.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.073+0000 7f6a02000700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f69e806e630 0x7f69e8070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:32.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.074+0000 7f6a02000700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f69e806e630 0x7f69e8070ae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f69f000b3b0 tx=0x7f69f0009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:32.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.077+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f69f405b110 con 0x7f6a04066ee0 2024-04-16T11:43:32.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83915 2024-04-16T11:43:32.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 12, success, outsize: 120 2024-04-16T11:43:32.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83997 2024-04-16T11:43:32.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 14, success, outsize: 120 2024-04-16T11:43:32.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84082 2024-04-16T11:43:32.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 16, success, outsize: 120 2024-04-16T11:43:32.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84173 2024-04-16T11:43:32.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 18, success, outsize: 120 2024-04-16T11:43:32.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84261 2024-04-16T11:43:32.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 20, success, outsize: 120 2024-04-16T11:43:32.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 22, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84343 2024-04-16T11:43:32.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 22, success, outsize: 120 2024-04-16T11:43:32.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 24, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84428 2024-04-16T11:43:32.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 24, success, outsize: 120 2024-04-16T11:43:32.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.454+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a04115e20 con 0x7f6a04066ee0 2024-04-16T11:43:32.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.455+0000 7f69faa00700 1 -- 172.21.15.44:0/3555880223 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f69f405aca0 con 0x7f6a04066ee0 2024-04-16T11:43:32.453 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:43:32.454 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:43:32.455 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T11:43:32.455 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:43:32.455 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:43:32.455 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:43:32.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.458+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f69e806e630 msgr2=0x7f69e8070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.458+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f69e806e630 0x7f69e8070ae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f69f000b3b0 tx=0x7f69f0009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:32.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.458+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 msgr2=0x7f6a0410fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.458+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 0x7f6a0410fc10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f69f400c800 tx=0x7f69f4004470 comp rx=0 tx=0).stop 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 shutdown_connections 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f69e806e630 0x7f69e8070ae0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a04066ee0 0x7f6a0410fc10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 --2- 172.21.15.44:0/3555880223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a040fffb0 0x7f6a04110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 >> 172.21.15.44:0/3555880223 conn(0x7f6a040f7480 msgr2=0x7f6a040fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 shutdown_connections 2024-04-16T11:43:32.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.459+0000 7f6a09c00700 1 -- 172.21.15.44:0/3555880223 wait complete. 2024-04-16T11:43:32.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84521 2024-04-16T11:43:32.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 26, success, outsize: 120 2024-04-16T11:43:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 28, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84610 2024-04-16T11:43:32.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 28, success, outsize: 120 2024-04-16T11:43:32.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84692 2024-04-16T11:43:32.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 30, success, outsize: 120 2024-04-16T11:43:32.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 32, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84777 2024-04-16T11:43:32.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 32, success, outsize: 120 2024-04-16T11:43:32.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 34, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84868 2024-04-16T11:43:32.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 34, success, outsize: 120 2024-04-16T11:43:32.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84956 2024-04-16T11:43:32.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 36, success, outsize: 120 2024-04-16T11:43:32.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 38, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85038 2024-04-16T11:43:32.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 38, success, outsize: 120 2024-04-16T11:43:32.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85123 2024-04-16T11:43:32.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 40, success, outsize: 120 2024-04-16T11:43:32.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85214 2024-04-16T11:43:32.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 42, success, outsize: 120 2024-04-16T11:43:32.588 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 44, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85303 2024-04-16T11:43:32.588 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 44, success, outsize: 120 2024-04-16T11:43:32.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 46, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85395 2024-04-16T11:43:32.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 46, success, outsize: 120 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 -- 172.21.15.44:0/3602278530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 msgr2=0x7f02a8070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 --2- 172.21.15.44:0/3602278530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a8070d00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0294004940 tx=0x7f0294009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 -- 172.21.15.44:0/3602278530 shutdown_connections 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 --2- 172.21.15.44:0/3602278530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a8070d00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 --2- 172.21.15.44:0/3602278530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02a8071e50 0x7f02a8070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 -- 172.21.15.44:0/3602278530 >> 172.21.15.44:0/3602278530 conn(0x7f02a80f8540 msgr2=0x7f02a80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 -- 172.21.15.44:0/3602278530 shutdown_connections 2024-04-16T11:43:32.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.606+0000 7f02af000700 1 -- 172.21.15.44:0/3602278530 wait complete. 2024-04-16T11:43:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.607+0000 7f02af000700 1 Processor -- start 2024-04-16T11:43:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.607+0000 7f02af000700 1 -- start start 2024-04-16T11:43:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.607+0000 7f02af000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.607+0000 7f02af000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02a8071e50 0x7f02a8110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.607+0000 7f02af000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8110790 con 0x7f02a8070890 2024-04-16T11:43:32.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.607+0000 7f02af000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8110fe0 con 0x7f02a8071e50 2024-04-16T11:43:32.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:32.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a810fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41278/0 (socket says 172.21.15.44:41278) 2024-04-16T11:43:32.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7e00700 1 -- 172.21.15.44:0/589439983 learned_addr learned my addr 172.21.15.44:0/589439983 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7400700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02a8071e50 0x7f02a8110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7e00700 1 -- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02a8071e50 msgr2=0x7f02a8110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7e00700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02a8071e50 0x7f02a8110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.608+0000 7f02a7e00700 1 -- 172.21.15.44:0/589439983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294005040 con 0x7f02a8070890 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.609+0000 7f02a7e00700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a810fc50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f029800c8b0 tx=0x7f0298004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.609+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f029800a760 con 0x7f02a8070890 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.609+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f029800ad90 con 0x7f02a8070890 2024-04-16T11:43:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.609+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a81112c0 con 0x7f02a8070890 2024-04-16T11:43:32.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.609+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02980113a0 con 0x7f02a8070890 2024-04-16T11:43:32.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.609+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a8111800 con 0x7f02a8070890 2024-04-16T11:43:32.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.611+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a805f5f0 con 0x7f02a8070890 2024-04-16T11:43:32.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.612+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f0298012400 con 0x7f02a8070890 2024-04-16T11:43:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.612+0000 7f029fe00700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f028c06e620 0x7f028c070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.613+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f029808de30 con 0x7f02a8070890 2024-04-16T11:43:32.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.613+0000 7f02a7400700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f028c06e620 0x7f028c070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:32.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.615+0000 7f02a7400700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f028c06e620 0x7f028c070ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0294004f90 tx=0x7f0294009e30 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:32.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.618+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f029805b0a0 con 0x7f02a8070890 2024-04-16T11:43:32.617 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85489 2024-04-16T11:43:32.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 48, success, outsize: 120 2024-04-16T11:43:32.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 50, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85580 2024-04-16T11:43:32.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 50, success, outsize: 120 2024-04-16T11:43:32.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 52, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85668 2024-04-16T11:43:32.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 52, success, outsize: 120 2024-04-16T11:43:32.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 54, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85750 2024-04-16T11:43:32.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 54, success, outsize: 120 2024-04-16T11:43:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 56, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85835 2024-04-16T11:43:32.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 56, success, outsize: 120 2024-04-16T11:43:32.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 58, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85926 2024-04-16T11:43:32.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 58, success, outsize: 120 2024-04-16T11:43:32.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 60, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86014 2024-04-16T11:43:32.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 60, success, outsize: 120 2024-04-16T11:43:32.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:32 smithi044 ceph-mon[28731]: pgmap v40: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:32.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:32 smithi044 ceph-mon[28731]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:32.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:32 smithi044 ceph-mon[28731]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:32.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:32 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:32.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:32 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/3555880223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:43:32.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 62, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86096 2024-04-16T11:43:32.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 62, success, outsize: 120 2024-04-16T11:43:32.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 64, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86181 2024-04-16T11:43:32.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 64, success, outsize: 120 2024-04-16T11:43:32.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 66, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86272 2024-04-16T11:43:32.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 66, success, outsize: 120 2024-04-16T11:43:32.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:32 smithi045 ceph-mon[36308]: pgmap v40: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:32.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:32 smithi045 ceph-mon[36308]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:32.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:32 smithi045 ceph-mon[36308]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:32.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:32 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:32.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:32 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3555880223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:43:32.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.923+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f02a8107ae0 con 0x7f02a8070890 2024-04-16T11:43:32.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.923+0000 7f029fe00700 1 -- 172.21.15.44:0/589439983 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7f029805ac30 con 0x7f02a8070890 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:43:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:43:32.923 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:43:32.924 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f028c06e620 msgr2=0x7f028c070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f028c06e620 0x7f028c070ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0294004f90 tx=0x7f0294009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 msgr2=0x7f02a810fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a810fc50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f029800c8b0 tx=0x7f0298004470 comp rx=0 tx=0).stop 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 shutdown_connections 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f028c06e620 0x7f028c070ad0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02a8070890 0x7f02a810fc50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 --2- 172.21.15.44:0/589439983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02a8071e50 0x7f02a8110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.926+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 >> 172.21.15.44:0/589439983 conn(0x7f02a80f8540 msgr2=0x7f02a80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:32.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.927+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 shutdown_connections 2024-04-16T11:43:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:32.927+0000 7f02af000700 1 -- 172.21.15.44:0/589439983 wait complete. 2024-04-16T11:43:32.926 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:43:33.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.086+0000 7f0c7aa00700 1 -- 172.21.15.44:0/726974164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 msgr2=0x7f0c740ffa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.086+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/726974164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c740ffa90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0c64004940 tx=0x7f0c64009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.086+0000 7f0c7aa00700 1 -- 172.21.15.44:0/726974164 shutdown_connections 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.086+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/726974164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740fffd0 0x7f0c741023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.086+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/726974164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c740ffa90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.086+0000 7f0c7aa00700 1 -- 172.21.15.44:0/726974164 >> 172.21.15.44:0/726974164 conn(0x7f0c740f74c0 msgr2=0x7f0c740f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.087+0000 7f0c7aa00700 1 -- 172.21.15.44:0/726974164 shutdown_connections 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.087+0000 7f0c7aa00700 1 -- 172.21.15.44:0/726974164 wait complete. 2024-04-16T11:43:33.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.088+0000 7f0c7aa00700 1 Processor -- start 2024-04-16T11:43:33.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.088+0000 7f0c7aa00700 1 -- start start 2024-04-16T11:43:33.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.088+0000 7f0c7aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c7410fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:33.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.088+0000 7f0c7aa00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740fffd0 0x7f0c74110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:33.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.088+0000 7f0c7aa00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74110730 con 0x7f0c74066290 2024-04-16T11:43:33.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.088+0000 7f0c7aa00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74110f80 con 0x7f0c740fffd0 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c7410fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c7410fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41288/0 (socket says 172.21.15.44:41288) 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73e00700 1 -- 172.21.15.44:0/3470842588 learned_addr learned my addr 172.21.15.44:0/3470842588 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73400700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740fffd0 0x7f0c74110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73e00700 1 -- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740fffd0 msgr2=0x7f0c74110130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73e00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740fffd0 0x7f0c74110130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.089+0000 7f0c73e00700 1 -- 172.21.15.44:0/3470842588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64005040 con 0x7f0c74066290 2024-04-16T11:43:33.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.090+0000 7f0c73e00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c7410fbf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0c64004090 tx=0x7f0c64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:33.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.090+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c64021070 con 0x7f0c74066290 2024-04-16T11:43:33.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.090+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c6401c400 con 0x7f0c74066290 2024-04-16T11:43:33.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.090+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c6400e070 con 0x7f0c74066290 2024-04-16T11:43:33.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.090+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c74111200 con 0x7f0c74066290 2024-04-16T11:43:33.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.090+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741116f0 con 0x7f0c74066290 2024-04-16T11:43:33.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.092+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7405f5f0 con 0x7f0c74066290 2024-04-16T11:43:33.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.093+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f0c6401ca20 con 0x7f0c74066290 2024-04-16T11:43:33.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.094+0000 7f0c6be00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0c5806e620 0x7f0c58070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:33.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.094+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f0c6408f200 con 0x7f0c74066290 2024-04-16T11:43:33.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.094+0000 7f0c73400700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0c5806e620 0x7f0c58070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:33.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.095+0000 7f0c73400700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0c5806e620 0x7f0c58070ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c60004600 tx=0x7f0c6000b040 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:33.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.098+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f0c6405c340 con 0x7f0c74066290 2024-04-16T11:43:33.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.337+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 --> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c74111a20 con 0x7f0c5806e620 2024-04-16T11:43:33.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.340+0000 7f0c6be00700 1 -- 172.21.15.44:0/3470842588 <== mgr.24393 v2:172.21.15.45:6828/1075678648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f0c74111a20 con 0x7f0c5806e620 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:43:33.339 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:43:33.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0c5806e620 msgr2=0x7f0c58070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0c5806e620 0x7f0c58070ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c60004600 tx=0x7f0c6000b040 comp rx=0 tx=0).stop 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 msgr2=0x7f0c7410fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c7410fbf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0c64004090 tx=0x7f0c64009e30 comp rx=0 tx=0).stop 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 shutdown_connections 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7f0c5806e620 0x7f0c58070ad0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c74066290 0x7f0c7410fbf0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 --2- 172.21.15.44:0/3470842588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740fffd0 0x7f0c74110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 >> 172.21.15.44:0/3470842588 conn(0x7f0c740f74c0 msgr2=0x7f0c741003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 shutdown_connections 2024-04-16T11:43:33.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.343+0000 7f0c7aa00700 1 -- 172.21.15.44:0/3470842588 wait complete. 2024-04-16T11:43:33.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.497+0000 7fd4b7c00700 1 -- 172.21.15.44:0/3603779172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 msgr2=0x7fd4b0103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.497+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/3603779172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0103810 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009ce0 tx=0x7fd4a4004d10 comp rx=0 tx=0).stop 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.498+0000 7fd4b7c00700 1 -- 172.21.15.44:0/3603779172 shutdown_connections 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.498+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/3603779172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 0x7fd4b0104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.498+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/3603779172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0103810 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.498+0000 7fd4b7c00700 1 -- 172.21.15.44:0/3603779172 >> 172.21.15.44:0/3603779172 conn(0x7fd4b0072590 msgr2=0x7fd4b00749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.498+0000 7fd4b7c00700 1 -- 172.21.15.44:0/3603779172 shutdown_connections 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.498+0000 7fd4b7c00700 1 -- 172.21.15.44:0/3603779172 wait complete. 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b7c00700 1 Processor -- start 2024-04-16T11:43:33.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b7c00700 1 -- start start 2024-04-16T11:43:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b7c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0114160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b7c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 0x7fd4b01146a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b7c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0114d30 con 0x7fd4b0103400 2024-04-16T11:43:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b7c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0115520 con 0x7fd4b01046b0 2024-04-16T11:43:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0114160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0114160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41308/0 (socket says 172.21.15.44:41308) 2024-04-16T11:43:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.499+0000 7fd4b5200700 1 -- 172.21.15.44:0/240505510 learned_addr learned my addr 172.21.15.44:0/240505510 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4afe00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 0x7fd4b01146a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4b5200700 1 -- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 msgr2=0x7fd4b01146a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4b5200700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 0x7fd4b01146a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4b5200700 1 -- 172.21.15.44:0/240505510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a4009940 con 0x7fd4b0103400 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4afe00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 0x7fd4b01146a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4b5200700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0114160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4004740 tx=0x7fd4a4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.500+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a400e040 con 0x7fd4b0103400 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.501+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b01157a0 con 0x7fd4b0103400 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.501+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4a40128e0 con 0x7fd4b0103400 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.501+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a401ca80 con 0x7fd4b0103400 2024-04-16T11:43:33.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.501+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b0115d60 con 0x7fd4b0103400 2024-04-16T11:43:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.502+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b005f5f0 con 0x7fd4b0103400 2024-04-16T11:43:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.503+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fd4a401cc70 con 0x7fd4b0103400 2024-04-16T11:43:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.504+0000 7fd4acc00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fd49806e620 0x7fd498070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:43:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.504+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fd4a408fd30 con 0x7fd4b0103400 2024-04-16T11:43:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.504+0000 7fd4afe00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fd49806e620 0x7fd498070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:43:33.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.506+0000 7fd4afe00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fd49806e620 0x7fd498070ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006fd0 tx=0x7fd49c008040 comp rx=0 tx=0).ready entity=mgr.24393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:43:33.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.509+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fd4a405cd50 con 0x7fd4b0103400 2024-04-16T11:43:33.669 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:33 smithi044 ceph-mon[28731]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:33.669 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:33 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/589439983' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:43:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:33 smithi045 ceph-mon[36308]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:33 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/589439983' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:43:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.838+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd4b010bf70 con 0x7fd4b0103400 2024-04-16T11:43:33.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.839+0000 7fd4acc00700 1 -- 172.21.15.44:0/240505510 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fd4a405c8e0 con 0x7fd4b0103400 2024-04-16T11:43:33.837 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.841+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fd49806e620 msgr2=0x7fd498070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.841+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fd49806e620 0x7fd498070ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006fd0 tx=0x7fd49c008040 comp rx=0 tx=0).stop 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.841+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 msgr2=0x7fd4b0114160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.841+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0114160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4004740 tx=0x7fd4a4004d10 comp rx=0 tx=0).stop 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 shutdown_connections 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:6828/1075678648,v1:172.21.15.45:6829/1075678648] conn(0x7fd49806e620 0x7fd498070ad0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4b0103400 0x7fd4b0114160 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 --2- 172.21.15.44:0/240505510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4b01046b0 0x7fd4b01146a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 >> 172.21.15.44:0/240505510 conn(0x7fd4b0072590 msgr2=0x7fd4b0073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 shutdown_connections 2024-04-16T11:43:33.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:43:33.842+0000 7fd4b7c00700 1 -- 172.21.15.44:0/240505510 wait complete. 2024-04-16T11:43:34.758 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: pgmap v41: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:34.758 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/240505510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr fail", "who": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr fail", "who": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: Activating manager daemon smithi044.sczobv 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd='[{"prefix": "mgr fail", "who": "smithi045.sdlwif"}]': finished 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: mgrmap e26: smithi044.sczobv(active, starting, since 0.0160141s) 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:43:34.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:43:34.760 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:34.761 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:34.761 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:43:34.761 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:34 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: pgmap v41: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/240505510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 172.21.15.45:0/3244294602' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr fail", "who": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "mgr fail", "who": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: Activating manager daemon smithi044.sczobv 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.24393 ' entity='mgr.smithi045.sdlwif' cmd='[{"prefix": "mgr fail", "who": "smithi045.sdlwif"}]': finished 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: mgrmap e26: smithi044.sczobv(active, starting, since 0.0160141s) 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:43:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:43:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:34 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:43:35.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:35 smithi045 ceph-mon[36308]: [16/Apr/2024:11:43:34] ENGINE Bus STARTING 2024-04-16T11:43:35.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:35 smithi045 ceph-mon[36308]: mgrmap e27: smithi044.sczobv(active, since 1.02116s) 2024-04-16T11:43:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:35 smithi044 ceph-mon[28731]: [16/Apr/2024:11:43:34] ENGINE Bus STARTING 2024-04-16T11:43:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:35 smithi044 ceph-mon[28731]: mgrmap e27: smithi044.sczobv(active, since 1.02116s) 2024-04-16T11:43:36.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:36 smithi045 ceph-mon[36308]: [16/Apr/2024:11:43:34] ENGINE Serving on http://172.21.15.44:8765 2024-04-16T11:43:36.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:36 smithi045 ceph-mon[36308]: [16/Apr/2024:11:43:34] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:43:36.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:36 smithi045 ceph-mon[36308]: [16/Apr/2024:11:43:34] ENGINE Bus STARTED 2024-04-16T11:43:36.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:36 smithi045 ceph-mon[36308]: pgmap v3: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:36.861 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:36 smithi044 ceph-mon[28731]: [16/Apr/2024:11:43:34] ENGINE Serving on http://172.21.15.44:8765 2024-04-16T11:43:36.861 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:36 smithi044 ceph-mon[28731]: [16/Apr/2024:11:43:34] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:43:36.861 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:36 smithi044 ceph-mon[28731]: [16/Apr/2024:11:43:34] ENGINE Bus STARTED 2024-04-16T11:43:36.861 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:36 smithi044 ceph-mon[28731]: pgmap v3: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:37.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:37 smithi045 ceph-mon[36308]: pgmap v4: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:37.885 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:37 smithi044 ceph-mon[28731]: pgmap v4: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:38.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:38 smithi045 ceph-mon[36308]: mgrmap e28: smithi044.sczobv(active, since 3s) 2024-04-16T11:43:38.868 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:38 smithi044 ceph-mon[28731]: mgrmap e28: smithi044.sczobv(active, since 3s) 2024-04-16T11:43:39.805 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:39 smithi044 ceph-mon[28731]: pgmap v5: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:39.805 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:39 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:39.805 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:39 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:39.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:39 smithi045 ceph-mon[36308]: pgmap v5: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:39.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:39 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:39.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:39 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:41.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:41 smithi045 ceph-mon[36308]: pgmap v6: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:41 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:41 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:41.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:41 smithi044 ceph-mon[28731]: pgmap v6: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:41.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:41 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:41.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:41 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:42.878 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:42 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:42.879 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:42 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:43.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:42 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:43.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:42 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:43.711 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:43 smithi044 ceph-mon[28731]: pgmap v7: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T11:43:43.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:43 smithi045 ceph-mon[36308]: pgmap v7: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:43:44.947 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:44 smithi044 ceph-mon[28731]: pgmap v8: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/640069700' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:43:45.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:44 smithi045 ceph-mon[36308]: pgmap v8: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T11:43:46.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:45 smithi045 ceph-mon[36308]: mgrmap e29: smithi044.sczobv(active, since 10s), standbys: smithi045.sdlwif 2024-04-16T11:43:46.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:45 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:43:46.158 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:45 smithi044 ceph-mon[28731]: mgrmap e29: smithi044.sczobv(active, since 10s), standbys: smithi045.sdlwif 2024-04-16T11:43:46.158 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:45 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:43:47.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:46 smithi045 ceph-mon[36308]: pgmap v9: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T11:43:47.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:46 smithi044 ceph-mon[28731]: pgmap v9: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T11:43:48.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 68, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87717 2024-04-16T11:43:48.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 68, success, outsize: 120 2024-04-16T11:43:48.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 70, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87814 2024-04-16T11:43:48.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 70, success, outsize: 120 2024-04-16T11:43:48.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 72, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87914 2024-04-16T11:43:48.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 72, success, outsize: 120 2024-04-16T11:43:48.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 74, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88020 2024-04-16T11:43:48.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 74, success, outsize: 120 2024-04-16T11:43:48.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 76, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88123 2024-04-16T11:43:48.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 76, success, outsize: 120 2024-04-16T11:43:48.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 78, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88220 2024-04-16T11:43:48.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 78, success, outsize: 120 2024-04-16T11:43:48.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 80, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88320 2024-04-16T11:43:48.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 80, success, outsize: 120 2024-04-16T11:43:48.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 82, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88426 2024-04-16T11:43:48.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 82, success, outsize: 120 2024-04-16T11:43:48.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 84, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88529 2024-04-16T11:43:48.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 84, success, outsize: 120 2024-04-16T11:43:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 86, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88626 2024-04-16T11:43:48.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 86, success, outsize: 120 2024-04-16T11:43:48.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 88, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88726 2024-04-16T11:43:48.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 88, success, outsize: 120 2024-04-16T11:43:48.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 90, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88832 2024-04-16T11:43:48.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 90, success, outsize: 120 2024-04-16T11:43:48.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 92, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88935 2024-04-16T11:43:48.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 92, success, outsize: 120 2024-04-16T11:43:48.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89032 2024-04-16T11:43:48.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 94, success, outsize: 120 2024-04-16T11:43:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 96, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89132 2024-04-16T11:43:48.459 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 96, success, outsize: 120 2024-04-16T11:43:48.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 98, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89238 2024-04-16T11:43:48.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 98, success, outsize: 120 2024-04-16T11:43:48.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 100, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89341 2024-04-16T11:43:48.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 100, success, outsize: 120 2024-04-16T11:43:48.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 102, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89438 2024-04-16T11:43:48.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 102, success, outsize: 120 2024-04-16T11:43:48.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 104, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89538 2024-04-16T11:43:48.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 104, success, outsize: 120 2024-04-16T11:43:48.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 106, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89644 2024-04-16T11:43:48.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 106, success, outsize: 120 2024-04-16T11:43:48.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 108, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89747 2024-04-16T11:43:48.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 108, success, outsize: 120 2024-04-16T11:43:48.636 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 110, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89844 2024-04-16T11:43:48.636 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 110, success, outsize: 120 2024-04-16T11:43:48.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 112, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89944 2024-04-16T11:43:48.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 112, success, outsize: 120 2024-04-16T11:43:48.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 114, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90050 2024-04-16T11:43:48.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 114, success, outsize: 120 2024-04-16T11:43:48.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 116, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90153 2024-04-16T11:43:48.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 116, success, outsize: 120 2024-04-16T11:43:48.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 118, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90250 2024-04-16T11:43:48.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 118, success, outsize: 120 2024-04-16T11:43:48.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 120, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90350 2024-04-16T11:43:48.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 120, success, outsize: 120 2024-04-16T11:43:48.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 122, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90456 2024-04-16T11:43:48.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 122, success, outsize: 120 2024-04-16T11:43:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:49 smithi045 ceph-mon[36308]: pgmap v10: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:43:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:49 smithi044 ceph-mon[28731]: pgmap v10: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:43:50.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:50 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:50.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:50 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:50.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:50 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:50.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:50 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:43:50.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:50 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:43:50.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:50 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:43:50.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:50 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:50.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:50 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:50 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:43:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:50 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:43:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:51 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:43:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:51 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:43:51.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:51 smithi045 ceph-mon[36308]: pgmap v11: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:43:51.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:51 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:43:51.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:51 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:43:51.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:51 smithi044 ceph-mon[28731]: pgmap v11: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:43:52.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:52 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:43:52.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:52 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:43:52.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:52 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:43:52.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:52 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:43:52.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:52 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:43:52.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:52 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:43:52.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:52 smithi044 ceph-mon[28731]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:43:52.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:52 smithi044 ceph-mon[28731]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:43:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:53 smithi045 ceph-mon[36308]: pgmap v12: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:43:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:53 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:43:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:53 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:43:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:53 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:53 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:53 smithi044 ceph-mon[28731]: pgmap v12: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:43:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:53 smithi044 ceph-mon[28731]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:43:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:53 smithi044 ceph-mon[28731]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:43:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:53 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:53 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:54.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:54 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:54.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:54 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:54.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:54 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:54.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:54 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:54.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:54 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:54.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:54 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:43:55.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:55 smithi044 ceph-mon[28731]: Reconfiguring prometheus.smithi044 (dependencies changed)... 2024-04-16T11:43:55.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:55 smithi044 ceph-mon[28731]: Reconfiguring daemon prometheus.smithi044 on smithi044 2024-04-16T11:43:55.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:55 smithi044 ceph-mon[28731]: pgmap v13: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:55 smithi045 ceph-mon[36308]: Reconfiguring prometheus.smithi044 (dependencies changed)... 2024-04-16T11:43:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:55 smithi045 ceph-mon[36308]: Reconfiguring daemon prometheus.smithi044 on smithi044 2024-04-16T11:43:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:55 smithi045 ceph-mon[36308]: pgmap v13: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:57 smithi044 ceph-mon[28731]: pgmap v14: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:57.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:57 smithi045 ceph-mon[36308]: pgmap v14: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:59.635 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:43:59 smithi044 ceph-mon[28731]: pgmap v15: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:43:59.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:43:59 smithi045 ceph-mon[36308]: pgmap v15: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:01.621 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:01 smithi044 ceph-mon[28731]: pgmap v16: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:01.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:01 smithi045 ceph-mon[36308]: pgmap v16: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:02 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:02 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:02 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:44:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:02 smithi044 ceph-mon[28731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:44:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:02 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:02 smithi044 ceph-mon[28731]: pgmap v17: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:03.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:02 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:03.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:02 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:03.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:02 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:44:03.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:02 smithi045 ceph-mon[36308]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:44:03.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:02 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:03.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:02 smithi045 ceph-mon[36308]: pgmap v17: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.015+0000 7fdc23a00700 1 -- 172.21.15.44:0/735230155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 msgr2=0x7fdc1c100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.015+0000 7fdc23a00700 1 --2- 172.21.15.44:0/735230155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c100740 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdc08004940 tx=0x7fdc08009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.016+0000 7fdc23a00700 1 -- 172.21.15.44:0/735230155 shutdown_connections 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.016+0000 7fdc23a00700 1 --2- 172.21.15.44:0/735230155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c100740 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.016+0000 7fdc23a00700 1 --2- 172.21.15.44:0/735230155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c0ff020 0x7fdc1c0ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.016+0000 7fdc23a00700 1 -- 172.21.15.44:0/735230155 >> 172.21.15.44:0/735230155 conn(0x7fdc1c0fa950 msgr2=0x7fdc1c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.016+0000 7fdc23a00700 1 -- 172.21.15.44:0/735230155 shutdown_connections 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.016+0000 7fdc23a00700 1 -- 172.21.15.44:0/735230155 wait complete. 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc23a00700 1 Processor -- start 2024-04-16T11:44:04.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc23a00700 1 -- start start 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc23a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c0ff020 0x7fdc1c10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc23a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc23a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c1106b0 con 0x7fdc1c1002d0 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc23a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c110820 con 0x7fdc1c0ff020 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.017+0000 7fdc1be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc1be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c1100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44900/0 (socket says 172.21.15.44:44900) 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc1be00700 1 -- 172.21.15.44:0/18691135 learned_addr learned my addr 172.21.15.44:0/18691135 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc21000700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c0ff020 0x7fdc1c10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc1be00700 1 -- 172.21.15.44:0/18691135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c0ff020 msgr2=0x7fdc1c10fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc1be00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c0ff020 0x7fdc1c10fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc1be00700 1 -- 172.21.15.44:0/18691135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc08005040 con 0x7fdc1c1002d0 2024-04-16T11:44:04.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.018+0000 7fdc1be00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c1100b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800b3b0 tx=0x7fdc08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:04.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.019+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc0800e040 con 0x7fdc1c1002d0 2024-04-16T11:44:04.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.019+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc08004290 con 0x7fdc1c1002d0 2024-04-16T11:44:04.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.019+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc0801c630 con 0x7fdc1c1002d0 2024-04-16T11:44:04.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.019+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c115220 con 0x7fdc1c1002d0 2024-04-16T11:44:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.019+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c1157e0 con 0x7fdc1c1002d0 2024-04-16T11:44:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.020+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c04c710 con 0x7fdc1c1002d0 2024-04-16T11:44:04.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.021+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fdc0800bbf0 con 0x7fdc1c1002d0 2024-04-16T11:44:04.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.022+0000 7fdc18c00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fdc0406e630 0x7fdc04070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.022+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdc0808f4a0 con 0x7fdc1c1002d0 2024-04-16T11:44:04.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.023+0000 7fdc21000700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fdc0406e630 0x7fdc04070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.023+0000 7fdc21000700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fdc0406e630 0x7fdc04070ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdc10004c40 tx=0x7fdc10009040 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:04.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.028+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fdc0805c520 con 0x7fdc1c1002d0 2024-04-16T11:44:04.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.286+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 --> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc1c1045f0 con 0x7fdc0406e630 2024-04-16T11:44:04.284 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:04 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:44:04.284 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:04 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:44:04.284 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:04 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.289+0000 7fdc18c00700 1 -- 172.21.15.44:0/18691135 <== mgr.14718 v2:172.21.15.44:6800/3774957652 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fdc1c1045f0 con 0x7fdc0406e630 2024-04-16T11:44:04.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fdc0406e630 msgr2=0x7fdc04070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fdc0406e630 0x7fdc04070ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdc10004c40 tx=0x7fdc10009040 comp rx=0 tx=0).stop 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 msgr2=0x7fdc1c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c1100b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800b3b0 tx=0x7fdc08009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 shutdown_connections 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fdc0406e630 0x7fdc04070ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc1c0ff020 0x7fdc1c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 --2- 172.21.15.44:0/18691135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc1c1002d0 0x7fdc1c1100b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 >> 172.21.15.44:0/18691135 conn(0x7fdc1c0fa950 msgr2=0x7fdc1c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 shutdown_connections 2024-04-16T11:44:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.293+0000 7fdc23a00700 1 -- 172.21.15.44:0/18691135 wait complete. 2024-04-16T11:44:04.308 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:44:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:04 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:44:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:04 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:44:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:04 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.456+0000 7efce5c00700 1 -- 172.21.15.44:0/1506042373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fcfa0 msgr2=0x7efce00fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.456+0000 7efce5c00700 1 --2- 172.21.15.44:0/1506042373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fcfa0 0x7efce00fd410 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7efcc8004940 tx=0x7efcc8009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.456+0000 7efce5c00700 1 -- 172.21.15.44:0/1506042373 shutdown_connections 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.456+0000 7efce5c00700 1 --2- 172.21.15.44:0/1506042373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fcfa0 0x7efce00fd410 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.456+0000 7efce5c00700 1 --2- 172.21.15.44:0/1506042373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efce00fbcf0 0x7efce00fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.457+0000 7efce5c00700 1 -- 172.21.15.44:0/1506042373 >> 172.21.15.44:0/1506042373 conn(0x7efce00f7450 msgr2=0x7efce00f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.457+0000 7efce5c00700 1 -- 172.21.15.44:0/1506042373 shutdown_connections 2024-04-16T11:44:04.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.457+0000 7efce5c00700 1 -- 172.21.15.44:0/1506042373 wait complete. 2024-04-16T11:44:04.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efce5c00700 1 Processor -- start 2024-04-16T11:44:04.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efce5c00700 1 -- start start 2024-04-16T11:44:04.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efce5c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 0x7efce0110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efce5c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efce00fcfa0 0x7efce0112c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efce5c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0066c50 con 0x7efce00fbcf0 2024-04-16T11:44:04.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efce5c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0066d90 con 0x7efce00fcfa0 2024-04-16T11:44:04.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efcdea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 0x7efce0110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efcdea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 0x7efce0110700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44906/0 (socket says 172.21.15.44:44906) 2024-04-16T11:44:04.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.458+0000 7efcdea00700 1 -- 172.21.15.44:0/4153199429 learned_addr learned my addr 172.21.15.44:0/4153199429 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:04.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.459+0000 7efcd6000700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efce00fcfa0 0x7efce0112c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.459+0000 7efcdea00700 1 -- 172.21.15.44:0/4153199429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efce00fcfa0 msgr2=0x7efce0112c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.459+0000 7efcdea00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efce00fcfa0 0x7efce0112c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.459+0000 7efcdea00700 1 -- 172.21.15.44:0/4153199429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8005040 con 0x7efce00fbcf0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.459+0000 7efcdea00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 0x7efce0110700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efcd0002a50 tx=0x7efcd00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.459+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcd0007cc0 con 0x7efce00fbcf0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.460+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcd000f040 con 0x7efce00fbcf0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.460+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcd0009640 con 0x7efce00fbcf0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.460+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce0113250 con 0x7efce00fbcf0 2024-04-16T11:44:04.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.460+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce0113810 con 0x7efce00fbcf0 2024-04-16T11:44:04.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.461+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce005f5f0 con 0x7efce00fbcf0 2024-04-16T11:44:04.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.462+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7efcd00097a0 con 0x7efce00fbcf0 2024-04-16T11:44:04.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.463+0000 7efcd7400700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7efcc006e620 0x7efcc0070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.463+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7efcd008de80 con 0x7efce00fbcf0 2024-04-16T11:44:04.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.463+0000 7efcd6000700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7efcc006e620 0x7efcc0070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.464+0000 7efcd6000700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7efcc006e620 0x7efcc0070ad0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efcc800b3b0 tx=0x7efcc8009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:04.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.468+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7efcd0057330 con 0x7efce00fbcf0 2024-04-16T11:44:04.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.718+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 --> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efce01012e0 con 0x7efcc006e620 2024-04-16T11:44:04.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.723+0000 7efcd7400700 1 -- 172.21.15.44:0/4153199429 <== mgr.14718 v2:172.21.15.44:6800/3774957652 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7efce01012e0 con 0x7efcc006e620 2024-04-16T11:44:04.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7efcc006e620 msgr2=0x7efcc0070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7efcc006e620 0x7efcc0070ad0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efcc800b3b0 tx=0x7efcc8009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:04.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 msgr2=0x7efce0110700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 0x7efce0110700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efcd0002a50 tx=0x7efcd00046b0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 shutdown_connections 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7efcc006e620 0x7efcc0070ad0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efce00fbcf0 0x7efce0110700 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 --2- 172.21.15.44:0/4153199429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efce00fcfa0 0x7efce0112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.726+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 >> 172.21.15.44:0/4153199429 conn(0x7efce00f7450 msgr2=0x7efce00f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.727+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 shutdown_connections 2024-04-16T11:44:04.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.727+0000 7efce5c00700 1 -- 172.21.15.44:0/4153199429 wait complete. 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.873+0000 7f937e200700 1 -- 172.21.15.44:0/1101750072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 msgr2=0x7f9378070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.873+0000 7f937e200700 1 --2- 172.21.15.44:0/1101750072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f9378070e10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9360004880 tx=0x7f9360009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.874+0000 7f937e200700 1 -- 172.21.15.44:0/1101750072 shutdown_connections 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.874+0000 7f937e200700 1 --2- 172.21.15.44:0/1101750072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f9378070e10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.874+0000 7f937e200700 1 --2- 172.21.15.44:0/1101750072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9378071f50 0x7f9378070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.874+0000 7f937e200700 1 -- 172.21.15.44:0/1101750072 >> 172.21.15.44:0/1101750072 conn(0x7f93780f8530 msgr2=0x7f93780fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.874+0000 7f937e200700 1 -- 172.21.15.44:0/1101750072 shutdown_connections 2024-04-16T11:44:04.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.874+0000 7f937e200700 1 -- 172.21.15.44:0/1101750072 wait complete. 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f937e200700 1 Processor -- start 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f937e200700 1 -- start start 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f937e200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f937810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f937e200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9378071f50 0x7f9378110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f937e200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9378110780 con 0x7f93780709a0 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f937e200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9378110fd0 con 0x7f9378071f50 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f9377400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f937810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f9377400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f937810fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44922/0 (socket says 172.21.15.44:44922) 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f9377400700 1 -- 172.21.15.44:0/822901561 learned_addr learned my addr 172.21.15.44:0/822901561 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.875+0000 7f936ea00700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9378071f50 0x7f9378110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.876+0000 7f9377400700 1 -- 172.21.15.44:0/822901561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9378071f50 msgr2=0x7f9378110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:04.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.876+0000 7f9377400700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9378071f50 0x7f9378110180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:04.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.876+0000 7f9377400700 1 -- 172.21.15.44:0/822901561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9368005040 con 0x7f93780709a0 2024-04-16T11:44:04.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.876+0000 7f9377400700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f937810fc40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f93680096b0 tx=0x7f9368009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:04.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.876+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f936800e840 con 0x7f93780709a0 2024-04-16T11:44:04.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.877+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f936800f040 con 0x7f93780709a0 2024-04-16T11:44:04.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.877+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9360005040 con 0x7f93780709a0 2024-04-16T11:44:04.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.877+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9368013dc0 con 0x7f93780709a0 2024-04-16T11:44:04.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.877+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9378111600 con 0x7f93780709a0 2024-04-16T11:44:04.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.878+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937805f5f0 con 0x7f93780709a0 2024-04-16T11:44:04.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.879+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f936800e9a0 con 0x7f93780709a0 2024-04-16T11:44:04.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.880+0000 7f936fe00700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f935c06e620 0x7f935c070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:04.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.880+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f936808e110 con 0x7f93780709a0 2024-04-16T11:44:04.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.881+0000 7f936ea00700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f935c06e620 0x7f935c070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:04.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.881+0000 7f936ea00700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f935c06e620 0x7f935c070ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9360009fd0 tx=0x7f9360009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:04.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:04.885+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f936805afd0 con 0x7f93780709a0 2024-04-16T11:44:05.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.125+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 --> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9378111990 con 0x7f935c06e620 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.156+0000 7f936fe00700 1 -- 172.21.15.44:0/822901561 <== mgr.14718 v2:172.21.15.44:6800/3774957652 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9378111990 con 0x7f935c06e620 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (73s) 23s ago 11m 14.0M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (11m) 23s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (10m) 26s ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (66s) 23s ago 11m 72.3M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:44:05.154 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (4m) 23s ago 4m 12.4M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (4m) 23s ago 4m 23.2M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (4m) 26s ago 4m 12.3M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (4m) 26s ago 4m 12.4M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (47s) 23s ago 13m 477M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283 running (2m) 26s ago 10m 74.8M - 19.0.0-2884-g218a6914 a4e4811815e4 de1d7adc1838 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (13m) 23s ago 13m 51.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (10m) 26s ago 10m 41.1M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (80s) 23s ago 11m 7390k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (55s) 26s ago 10m 7604k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (9m) 23s ago 9m 46.2M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (8m) 23s ago 8m 48.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (8m) 23s ago 8m 45.4M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (7m) 26s ago 7m 45.7M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (6m) 26s ago 6m 43.5M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (6m) 26s ago 6m 45.5M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:44:05.155 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 starting - - - - 2024-04-16T11:44:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f935c06e620 msgr2=0x7f935c070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f935c06e620 0x7f935c070ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9360009fd0 tx=0x7f9360009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 msgr2=0x7f937810fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f937810fc40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f93680096b0 tx=0x7f9368009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 shutdown_connections 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f935c06e620 0x7f935c070ad0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93780709a0 0x7f937810fc40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 --2- 172.21.15.44:0/822901561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9378071f50 0x7f9378110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.159+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 >> 172.21.15.44:0/822901561 conn(0x7f93780f8530 msgr2=0x7f93780f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.160+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 shutdown_connections 2024-04-16T11:44:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.160+0000 7f937e200700 1 -- 172.21.15.44:0/822901561 wait complete. 2024-04-16T11:44:05.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 -- 172.21.15.44:0/2379199026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c00ff020 msgr2=0x7f03c00ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 --2- 172.21.15.44:0/2379199026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c00ff020 0x7f03c00ff430 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f03b0004940 tx=0x7f03b0009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:05.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 -- 172.21.15.44:0/2379199026 shutdown_connections 2024-04-16T11:44:05.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 --2- 172.21.15.44:0/2379199026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c01002d0 0x7f03c0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 --2- 172.21.15.44:0/2379199026 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c00ff020 0x7f03c00ff430 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 -- 172.21.15.44:0/2379199026 >> 172.21.15.44:0/2379199026 conn(0x7f03c00fa950 msgr2=0x7f03c00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:05.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 -- 172.21.15.44:0/2379199026 shutdown_connections 2024-04-16T11:44:05.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.309+0000 7f03c5400700 1 -- 172.21.15.44:0/2379199026 wait complete. 2024-04-16T11:44:05.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.310+0000 7f03c5400700 1 Processor -- start 2024-04-16T11:44:05.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.310+0000 7f03c5400700 1 -- start start 2024-04-16T11:44:05.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03c5400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ff020 0x7f03c010fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:05.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03c5400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 0x7f03c0110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:05.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03c5400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0110620 con 0x7f03c01002d0 2024-04-16T11:44:05.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03c5400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0110790 con 0x7f03c00ff020 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03bde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 0x7f03c0110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03bde00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 0x7f03c0110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44938/0 (socket says 172.21.15.44:44938) 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03bde00700 1 -- 172.21.15.44:0/2304695148 learned_addr learned my addr 172.21.15.44:0/2304695148 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03be800700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ff020 0x7f03c010fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03bde00700 1 -- 172.21.15.44:0/2304695148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ff020 msgr2=0x7f03c010fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03bde00700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ff020 0x7f03c010fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.311+0000 7f03bde00700 1 -- 172.21.15.44:0/2304695148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b0005040 con 0x7f03c01002d0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.312+0000 7f03bde00700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 0x7f03c0110020 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f03ac002a80 tx=0x7f03ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.312+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03ac007cc0 con 0x7f03c01002d0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.312+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03ac00f040 con 0x7f03c01002d0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.312+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c01151f0 con 0x7f03c01002d0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.312+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03ac0096f0 con 0x7f03c01002d0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.312+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c01157b0 con 0x7f03c01002d0 2024-04-16T11:44:05.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.313+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c004c710 con 0x7f03c01002d0 2024-04-16T11:44:05.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.314+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f03ac00d690 con 0x7f03c01002d0 2024-04-16T11:44:05.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.316+0000 7f03b6a00700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f03a406e630 0x7f03a4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:05.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.316+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f03ac08d8d0 con 0x7f03c01002d0 2024-04-16T11:44:05.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.316+0000 7f03be800700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f03a406e630 0x7f03a4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:05.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.317+0000 7f03be800700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f03a406e630 0x7f03a4070ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f03b0004f90 tx=0x7f03b0009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:05.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.320+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f03ac05a810 con 0x7f03c01002d0 2024-04-16T11:44:05.398 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:05 smithi045 ceph-mon[36308]: Upgrade: Updating mgr.smithi045.sdlwif 2024-04-16T11:44:05.398 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:05 smithi045 ceph-mon[36308]: Deploying daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:44:05.398 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:05 smithi045 ceph-mon[36308]: pgmap v18: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:05.398 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:05 smithi045 ceph-mon[36308]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:05.398 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:05 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:05.399 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:05 smithi045 ceph-mon[36308]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:05.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:05 smithi044 ceph-mon[28731]: Upgrade: Updating mgr.smithi045.sdlwif 2024-04-16T11:44:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:05 smithi044 ceph-mon[28731]: Deploying daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:44:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:05 smithi044 ceph-mon[28731]: pgmap v18: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:05 smithi044 ceph-mon[28731]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:05 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:05 smithi044 ceph-mon[28731]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.631+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f03c0115c30 con 0x7f03c01002d0 2024-04-16T11:44:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.632+0000 7f03b6a00700 1 -- 172.21.15.44:0/2304695148 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f03ac05a3a0 con 0x7f03c01002d0 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:44:05.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T11:44:05.632 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:44:05.632 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:44:05.632 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:44:05.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f03a406e630 msgr2=0x7f03a4070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f03a406e630 0x7f03a4070ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f03b0004f90 tx=0x7f03b0009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 msgr2=0x7f03c0110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 0x7f03c0110020 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f03ac002a80 tx=0x7f03ac0046b0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 shutdown_connections 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f03a406e630 0x7f03a4070ae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03c00ff020 0x7f03c010fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 --2- 172.21.15.44:0/2304695148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03c01002d0 0x7f03c0110020 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.635+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 >> 172.21.15.44:0/2304695148 conn(0x7f03c00fa950 msgr2=0x7f03c00fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.636+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 shutdown_connections 2024-04-16T11:44:05.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.636+0000 7f03c5400700 1 -- 172.21.15.44:0/2304695148 wait complete. 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 -- 172.21.15.44:0/1225558241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb40066fe0 msgr2=0x7fcb400ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 --2- 172.21.15.44:0/1225558241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb40066fe0 0x7fcb400ff9e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcb30004940 tx=0x7fcb30009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 -- 172.21.15.44:0/1225558241 shutdown_connections 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 --2- 172.21.15.44:0/1225558241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb400fff20 0x7fcb40102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 --2- 172.21.15.44:0/1225558241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb40066fe0 0x7fcb400ff9e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 -- 172.21.15.44:0/1225558241 >> 172.21.15.44:0/1225558241 conn(0x7fcb400f7450 msgr2=0x7fcb400f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 -- 172.21.15.44:0/1225558241 shutdown_connections 2024-04-16T11:44:05.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.792+0000 7fcb45600700 1 -- 172.21.15.44:0/1225558241 wait complete. 2024-04-16T11:44:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.793+0000 7fcb45600700 1 Processor -- start 2024-04-16T11:44:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.793+0000 7fcb45600700 1 -- start start 2024-04-16T11:44:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.793+0000 7fcb45600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb40066fe0 0x7fcb4010fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.793+0000 7fcb45600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 0x7fcb40110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.793+0000 7fcb45600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40110730 con 0x7fcb400fff20 2024-04-16T11:44:05.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.793+0000 7fcb45600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb401108a0 con 0x7fcb40066fe0 2024-04-16T11:44:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 0x7fcb40110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 0x7fcb40110130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44956/0 (socket says 172.21.15.44:44956) 2024-04-16T11:44:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 -- 172.21.15.44:0/2466780287 learned_addr learned my addr 172.21.15.44:0/2466780287 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3ea00700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb40066fe0 0x7fcb4010fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 -- 172.21.15.44:0/2466780287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb40066fe0 msgr2=0x7fcb4010fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:05.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb40066fe0 0x7fcb4010fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:05.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 -- 172.21.15.44:0/2466780287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30005040 con 0x7fcb400fff20 2024-04-16T11:44:05.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.794+0000 7fcb3e000700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 0x7fcb40110130 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c002a50 tx=0x7fcb2c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:05.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.795+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb2c007cc0 con 0x7fcb400fff20 2024-04-16T11:44:05.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.795+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb2c00f040 con 0x7fcb400fff20 2024-04-16T11:44:05.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.795+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb2c00db70 con 0x7fcb400fff20 2024-04-16T11:44:05.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.795+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb401152b0 con 0x7fcb400fff20 2024-04-16T11:44:05.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.795+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb40115870 con 0x7fcb400fff20 2024-04-16T11:44:05.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.796+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4004c710 con 0x7fcb400fff20 2024-04-16T11:44:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.797+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fcb2c00dd10 con 0x7fcb400fff20 2024-04-16T11:44:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.798+0000 7fcb36a00700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fcb2406e630 0x7fcb24070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.798+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fcb2c08d770 con 0x7fcb400fff20 2024-04-16T11:44:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.799+0000 7fcb3ea00700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fcb2406e630 0x7fcb24070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:05.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.800+0000 7fcb3ea00700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fcb2406e630 0x7fcb24070ae0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009fd0 tx=0x7fcb30009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:05.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:05.804+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fcb2c05a750 con 0x7fcb400fff20 2024-04-16T11:44:06.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.085+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcb40115cc0 con 0x7fcb400fff20 2024-04-16T11:44:06.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.086+0000 7fcb36a00700 1 -- 172.21.15.44:0/2466780287 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fcb2c05a2e0 con 0x7fcb400fff20 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:44:06.085 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:44:06.086 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:06.087 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:06.088 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:06.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fcb2406e630 msgr2=0x7fcb24070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fcb2406e630 0x7fcb24070ae0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009fd0 tx=0x7fcb30009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:06.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 msgr2=0x7fcb40110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 0x7fcb40110130 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c002a50 tx=0x7fcb2c0046b0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 shutdown_connections 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7fcb2406e630 0x7fcb24070ae0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb40066fe0 0x7fcb4010fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 --2- 172.21.15.44:0/2466780287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb400fff20 0x7fcb40110130 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 >> 172.21.15.44:0/2466780287 conn(0x7fcb400f7450 msgr2=0x7fcb400fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.089+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 shutdown_connections 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.090+0000 7fcb45600700 1 -- 172.21.15.44:0/2466780287 wait complete. 2024-04-16T11:44:06.089 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 -- 172.21.15.44:0/4011954832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c104520 msgr2=0x7f491c104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 --2- 172.21.15.44:0/4011954832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c104520 0x7f491c104990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4904004940 tx=0x7f4904009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 -- 172.21.15.44:0/4011954832 shutdown_connections 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 --2- 172.21.15.44:0/4011954832 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c104520 0x7f491c104990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 --2- 172.21.15.44:0/4011954832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c103270 0x7f491c103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 -- 172.21.15.44:0/4011954832 >> 172.21.15.44:0/4011954832 conn(0x7f491c072530 msgr2=0x7f491c074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 -- 172.21.15.44:0/4011954832 shutdown_connections 2024-04-16T11:44:06.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.245+0000 7f491be00700 1 -- 172.21.15.44:0/4011954832 wait complete. 2024-04-16T11:44:06.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.246+0000 7f491be00700 1 Processor -- start 2024-04-16T11:44:06.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.246+0000 7f491be00700 1 -- start start 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 0x7f491c113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c104520 0x7f491c114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491be00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c114a60 con 0x7f491c103270 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c1151a0 con 0x7f491c104520 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 0x7f491c113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 0x7f491c113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44972/0 (socket says 172.21.15.44:44972) 2024-04-16T11:44:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491aa00700 1 -- 172.21.15.44:0/329900437 learned_addr learned my addr 172.21.15.44:0/329900437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f4913e00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c104520 0x7f491c114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491aa00700 1 -- 172.21.15.44:0/329900437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c104520 msgr2=0x7f491c114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491aa00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c104520 0x7f491c114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f491aa00700 1 -- 172.21.15.44:0/329900437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904005040 con 0x7f491c103270 2024-04-16T11:44:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.247+0000 7f4913e00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c104520 0x7f491c114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.248+0000 7f491aa00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 0x7f491c113f20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f490c00c830 tx=0x7f490c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.248+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f490c00a760 con 0x7f491c103270 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.248+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f490c00ad90 con 0x7f491c103270 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.248+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491c115480 con 0x7f491c103270 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.248+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f490c0113a0 con 0x7f491c103270 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.248+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c115a40 con 0x7f491c103270 2024-04-16T11:44:06.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.250+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c0760a0 con 0x7f491c103270 2024-04-16T11:44:06.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.251+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f490c012400 con 0x7f491c103270 2024-04-16T11:44:06.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.252+0000 7f4912a00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f490006e630 0x7f4900070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:06.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.252+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f490c08e0b0 con 0x7f491c103270 2024-04-16T11:44:06.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.252+0000 7f4913e00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f490006e630 0x7f4900070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:06.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.253+0000 7f4913e00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f490006e630 0x7f4900070ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f490400b7a0 tx=0x7f4904009e30 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:06.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.257+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f490c05b1a0 con 0x7f491c103270 2024-04-16T11:44:06.256 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:06 smithi044 ceph-mon[28731]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:06.257 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:06 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2304695148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:06.257 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:06 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2466780287' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:44:06.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.499+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 --> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f491c115dd0 con 0x7f490006e630 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.501+0000 7f4912a00700 1 -- 172.21.15.44:0/329900437 <== mgr.14718 v2:172.21.15.44:6800/3774957652 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f491c115dd0 con 0x7f490006e630 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T11:44:06.499 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T11:44:06.500 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:44:06.500 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:44:06.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f490006e630 msgr2=0x7f4900070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f490006e630 0x7f4900070ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f490400b7a0 tx=0x7f4904009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:06.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 msgr2=0x7f491c113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 0x7f491c113f20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f490c00c830 tx=0x7f490c004470 comp rx=0 tx=0).stop 2024-04-16T11:44:06.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 shutdown_connections 2024-04-16T11:44:06.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f490006e630 0x7f4900070ae0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f491c103270 0x7f491c113f20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 --2- 172.21.15.44:0/329900437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f491c104520 0x7f491c114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 >> 172.21.15.44:0/329900437 conn(0x7f491c072530 msgr2=0x7f491c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:06.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 shutdown_connections 2024-04-16T11:44:06.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.504+0000 7f491be00700 1 -- 172.21.15.44:0/329900437 wait complete. 2024-04-16T11:44:06.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:06 smithi045 ceph-mon[36308]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:06 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2304695148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:06 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2466780287' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.655+0000 7f1f19c00700 1 -- 172.21.15.44:0/672999923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 msgr2=0x7f1f14072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.655+0000 7f1f19c00700 1 --2- 172.21.15.44:0/672999923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14072c30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1f04004940 tx=0x7f1f04009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.655+0000 7f1f19c00700 1 -- 172.21.15.44:0/672999923 shutdown_connections 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.655+0000 7f1f19c00700 1 --2- 172.21.15.44:0/672999923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f14073170 0x7f1f14073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.655+0000 7f1f19c00700 1 --2- 172.21.15.44:0/672999923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14072c30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.655+0000 7f1f19c00700 1 -- 172.21.15.44:0/672999923 >> 172.21.15.44:0/672999923 conn(0x7f1f140fa790 msgr2=0x7f1f140fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.656+0000 7f1f19c00700 1 -- 172.21.15.44:0/672999923 shutdown_connections 2024-04-16T11:44:06.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.656+0000 7f1f19c00700 1 -- 172.21.15.44:0/672999923 wait complete. 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.656+0000 7f1f19c00700 1 Processor -- start 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f19c00700 1 -- start start 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f19c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f19c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f14114390 0x7f1f14115100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f19c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14114850 con 0x7f1f14072820 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f19c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f141149c0 con 0x7f1f14114390 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f13e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f13e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44986/0 (socket says 172.21.15.44:44986) 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f13e00700 1 -- 172.21.15.44:0/2361360851 learned_addr learned my addr 172.21.15.44:0/2361360851 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.657+0000 7f1f13400700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f14114390 0x7f1f14115100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:06.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f13e00700 1 -- 172.21.15.44:0/2361360851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f14114390 msgr2=0x7f1f14115100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f13e00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f14114390 0x7f1f14115100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f13e00700 1 -- 172.21.15.44:0/2361360851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04005040 con 0x7f1f14072820 2024-04-16T11:44:06.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f13e00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14113e50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1f04000c00 tx=0x7f1f04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:06.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0400e030 con 0x7f1f14072820 2024-04-16T11:44:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f04013de0 con 0x7f1f14072820 2024-04-16T11:44:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0401ce50 con 0x7f1f14072820 2024-04-16T11:44:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.658+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f14115780 con 0x7f1f14072820 2024-04-16T11:44:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.659+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f14115cc0 con 0x7f1f14072820 2024-04-16T11:44:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.660+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f140fe290 con 0x7f1f14072820 2024-04-16T11:44:06.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.661+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 29) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f1f0401c400 con 0x7f1f14072820 2024-04-16T11:44:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.662+0000 7f1f0be00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f1ef806e630 0x7f1ef8070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.662+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1f0408fa80 con 0x7f1f14072820 2024-04-16T11:44:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.662+0000 7f1f13400700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f1ef806e630 0x7f1ef8070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.663+0000 7f1f13400700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f1ef806e630 0x7f1ef8070ae0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009b80 tx=0x7f1f0000b040 comp rx=0 tx=0).ready entity=mgr.14718 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:06.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.666+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f1f04058e90 con 0x7f1f14072820 2024-04-16T11:44:06.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.986+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1f14115910 con 0x7f1f14072820 2024-04-16T11:44:06.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.986+0000 7f1f0be00700 1 -- 172.21.15.44:0/2361360851 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1f14115910 con 0x7f1f14072820 2024-04-16T11:44:06.984 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:44:06.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f1ef806e630 msgr2=0x7f1ef8070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f1ef806e630 0x7f1ef8070ae0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009b80 tx=0x7f1f0000b040 comp rx=0 tx=0).stop 2024-04-16T11:44:06.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 msgr2=0x7f1f14113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14113e50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1f04000c00 tx=0x7f1f04009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 shutdown_connections 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f1ef806e630 0x7f1ef8070ae0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f14072820 0x7f1f14113e50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 --2- 172.21.15.44:0/2361360851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f14114390 0x7f1f14115100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 >> 172.21.15.44:0/2361360851 conn(0x7f1f140fa790 msgr2=0x7f1f140fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.989+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 shutdown_connections 2024-04-16T11:44:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:06.990+0000 7f1f19c00700 1 -- 172.21.15.44:0/2361360851 wait complete. 2024-04-16T11:44:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:07 smithi044 ceph-mon[28731]: pgmap v19: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:07 smithi044 ceph-mon[28731]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:07 smithi044 ceph-mon[28731]: from='client.? 172.21.15.44:0/2361360851' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:44:07.544 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:07 smithi045 ceph-mon[36308]: pgmap v19: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:07.544 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:07 smithi045 ceph-mon[36308]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:07.544 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:07 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2361360851' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:44:09.445 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:09 smithi045 ceph-mon[36308]: pgmap v20: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:09.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:09 smithi044 ceph-mon[28731]: pgmap v20: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:10.933 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:10 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:10.933 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:10 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:10.933 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:10 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:10.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:10 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:10.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:10 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:10.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:10 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:11.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:11 smithi044 ceph-mon[28731]: pgmap v21: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:11.987 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:11 smithi045 ceph-mon[36308]: pgmap v21: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:13.824 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:13 smithi044 ceph-mon[28731]: pgmap v22: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:14.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:13 smithi045 ceph-mon[36308]: pgmap v22: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:15.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:14 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:15.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:14 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:15.184 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:14 smithi044 ceph-mon[28731]: pgmap v23: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:14 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:14 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:14 smithi045 ceph-mon[36308]: pgmap v23: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:17.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:16 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:16 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:16 smithi045 ceph-mon[36308]: pgmap v24: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:17.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:16 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:16 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:16 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:16 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:16 smithi044 ceph-mon[28731]: pgmap v24: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:16 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:17.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:16 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: pgmap v25: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif restarted 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:44:19.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:19 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: pgmap v25: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: Standby manager daemon smithi045.sdlwif restarted 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:44:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:19 smithi044 ceph-mon[28731]: from='mgr.? 172.21.15.45:0/3053152846' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:44:19.839 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 124, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73849 2024-04-16T11:44:19.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 124, success, outsize: 120 2024-04-16T11:44:19.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 126, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73942 2024-04-16T11:44:19.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 126, success, outsize: 120 2024-04-16T11:44:19.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 128, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74038 2024-04-16T11:44:19.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 128, success, outsize: 120 2024-04-16T11:44:19.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 130, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74140 2024-04-16T11:44:19.916 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 130, success, outsize: 120 2024-04-16T11:44:19.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 132, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74239 2024-04-16T11:44:19.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 132, success, outsize: 120 2024-04-16T11:44:19.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 134, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74332 2024-04-16T11:44:19.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 134, success, outsize: 120 2024-04-16T11:44:20.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 136, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74428 2024-04-16T11:44:20.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 136, success, outsize: 120 2024-04-16T11:44:20.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 138, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74530 2024-04-16T11:44:20.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 138, success, outsize: 120 2024-04-16T11:44:20.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 140, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74629 2024-04-16T11:44:20.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 140, success, outsize: 120 2024-04-16T11:44:20.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 142, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74722 2024-04-16T11:44:20.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 142, success, outsize: 120 2024-04-16T11:44:20.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 144, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74818 2024-04-16T11:44:20.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 144, success, outsize: 120 2024-04-16T11:44:20.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 146, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74920 2024-04-16T11:44:20.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 146, success, outsize: 120 2024-04-16T11:44:20.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 148, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75019 2024-04-16T11:44:20.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 148, success, outsize: 120 2024-04-16T11:44:20.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 150, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75112 2024-04-16T11:44:20.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 150, success, outsize: 120 2024-04-16T11:44:20.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 152, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75208 2024-04-16T11:44:20.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 152, success, outsize: 120 2024-04-16T11:44:20.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 154, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75310 2024-04-16T11:44:20.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 154, success, outsize: 120 2024-04-16T11:44:20.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 156, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75409 2024-04-16T11:44:20.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 156, success, outsize: 120 2024-04-16T11:44:20.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 158, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75502 2024-04-16T11:44:20.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 158, success, outsize: 120 2024-04-16T11:44:20.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 160, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75598 2024-04-16T11:44:20.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 160, success, outsize: 120 2024-04-16T11:44:20.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 162, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75700 2024-04-16T11:44:20.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 162, success, outsize: 120 2024-04-16T11:44:20.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 164, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75799 2024-04-16T11:44:20.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 164, success, outsize: 120 2024-04-16T11:44:20.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 166, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75892 2024-04-16T11:44:20.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 166, success, outsize: 120 2024-04-16T11:44:20.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 168, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75988 2024-04-16T11:44:20.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 168, success, outsize: 120 2024-04-16T11:44:20.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 170, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76090 2024-04-16T11:44:20.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 170, success, outsize: 120 2024-04-16T11:44:20.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:20 smithi045 ceph-mon[36308]: mgrmap e30: smithi044.sczobv(active, since 45s), standbys: smithi045.sdlwif 2024-04-16T11:44:20.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:20 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:20.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 172, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76189 2024-04-16T11:44:20.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 172, success, outsize: 120 2024-04-16T11:44:20.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 174, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76282 2024-04-16T11:44:20.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 174, success, outsize: 120 2024-04-16T11:44:20.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 176, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76378 2024-04-16T11:44:20.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 176, success, outsize: 120 2024-04-16T11:44:20.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 178, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76480 2024-04-16T11:44:20.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 178, success, outsize: 120 2024-04-16T11:44:20.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:20 smithi044 ceph-mon[28731]: mgrmap e30: smithi044.sczobv(active, since 45s), standbys: smithi045.sdlwif 2024-04-16T11:44:20.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:20 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:21.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:21 smithi045 ceph-mon[36308]: pgmap v26: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:21.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:21 smithi044 ceph-mon[28731]: pgmap v26: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:22.962 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:22.962 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:22.962 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:22.962 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi044.sczobv"}]: dispatch 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi044.sczobv"}]': finished 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi045.sdlwif"}]: dispatch 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi045.sdlwif"}]': finished 2024-04-16T11:44:22.963 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:22 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi044.sczobv"}]: dispatch 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi044.sczobv"}]': finished 2024-04-16T11:44:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi045.sdlwif"}]: dispatch 2024-04-16T11:44:23.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi045.sdlwif"}]': finished 2024-04-16T11:44:23.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:22 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T11:44:23.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:23 smithi044 ceph-mon[28731]: Upgrade: Setting container_image for all mgr 2024-04-16T11:44:23.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:23 smithi044 ceph-mon[28731]: pgmap v27: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:24.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:23 smithi045 ceph-mon[36308]: Upgrade: Setting container_image for all mgr 2024-04-16T11:44:24.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:23 smithi045 ceph-mon[36308]: pgmap v27: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:25.467 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: pgmap v28: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:25.467 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: Upgrade: Updating mon.smithi044 2024-04-16T11:44:25.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:25.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:44:25.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:44:25.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:25.468 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:25 smithi044 ceph-mon[28731]: Deploying daemon mon.smithi044 on smithi044 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: pgmap v28: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: Upgrade: Updating mon.smithi044 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:25 smithi045 ceph-mon[36308]: Deploying daemon mon.smithi044 on smithi044 2024-04-16T11:44:27.319 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:27 smithi044 systemd[1]: Stopping Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:44:27.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:27 smithi044 ceph-mon[28731]: pgmap v29: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:27.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:27 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[28727]: 2024-04-16T11:44:27.444+0000 7f1128800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi044 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:44:27.579 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:27 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[28727]: 2024-04-16T11:44:27.444+0000 7f1128800700 -1 mon.smithi044@0(leader) e2 *** Got Signal Terminated *** 2024-04-16T11:44:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:27 smithi045 ceph-mon[36308]: pgmap v29: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:27.900 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:27 smithi044 podman[92846]: 2024-04-16 11:44:27.60820333 +0000 UTC m=+0.302267643 container died 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-16T11:44:28.161 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:28 smithi044 podman[92846]: 2024-04-16 11:44:28.146774634 +0000 UTC m=+0.840838937 container cleanup 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T11:44:28.161 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:28 smithi044 bash[92846]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044 2024-04-16T11:44:28.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:28 smithi044 podman[92871]: 2024-04-16 11:44:28.43996298 +0000 UTC m=+0.813794031 container remove 851468b7fc2b92db509432eb84463d3af1b3792b11f3307c3330655697c7f8fb (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5) 2024-04-16T11:44:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:28 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service: Deactivated successfully. 2024-04-16T11:44:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:28 smithi044 systemd[1]: Stopped Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:44:29.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:28 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service: Consumed 25.679s CPU time. 2024-04-16T11:44:29.665 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:29 smithi044 systemd[1]: Starting Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:44:29.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:29 smithi044 podman[93001]: 2024-04-16 11:44:29.5736375 +0000 UTC m=+0.035442243 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:44:29.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:29 smithi044 podman[93001]: 2024-04-16 11:44:29.683203838 +0000 UTC m=+0.145008621 container create 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-16T11:44:29.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:29 smithi044 podman[93001]: 2024-04-16 11:44:29.962507558 +0000 UTC m=+0.424312345 container init 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-16T11:44:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:29 smithi044 podman[93001]: 2024-04-16 11:44:29.974615846 +0000 UTC m=+0.436420619 container start 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-16T11:44:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: pidfile_write: ignore empty --pid-file 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: load: jerasure load: lrc 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: RocksDB version: 7.9.2 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Git sha 0 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: DB SUMMARY 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: DB Session ID: VMQIEY6SUG9EJ49WGK97 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: CURRENT file: CURRENT 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1147 Bytes 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi044/store.db dir, Total Num: 1, files: 000033.sst 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi044/store.db: 000031.log size: 1225992 ; 2024-04-16T11:44:30.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.error_if_exists: 0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.create_if_missing: 0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.paranoid_checks: 1 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.env: 0x55c51c73ccc0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.info_log: 0x55c5455a1ae0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.statistics: (nil) 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.use_fsync: 0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_log_file_size: 0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T11:44:30.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T11:44:30.230 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T11:44:30.230 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.allow_fallocate: 1 2024-04-16T11:44:30.230 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.use_direct_reads: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.db_log_dir: 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.wal_dir: 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T11:44:30.231 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.write_buffer_manager: 0x55c5455721e0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.unordered_write: 0 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T11:44:30.232 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.row_cache: None 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.wal_filter: None 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.two_write_queues: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.wal_compression: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.atomic_flush: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.log_readahead_size: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T11:44:30.233 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_background_jobs: 2 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_background_compactions: -1 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_subcompactions: 1 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T11:44:30.234 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_open_files: -1 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_background_flushes: -1 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Compression algorithms supported: 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kZSTD supported: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kXpressCompression supported: 0 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kZlibCompression supported: 1 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kSnappyCompression supported: 1 2024-04-16T11:44:30.235 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kLZ4Compression supported: 1 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: kBZip2Compression supported: 0 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi044/store.db/MANIFEST-000015 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.merge_operator: 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_filter: None 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c5455a1cc0) 2024-04-16T11:44:30.236 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: index_type: 0 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: data_block_index_type: 0 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: index_shortening: 1 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: checksum: 4 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: no_block_cache: 0 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache: 0x55c5455a5090 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache_options: 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: capacity : 536870912 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: num_shard_bits : 4 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: strict_capacity_limit : 0 2024-04-16T11:44:30.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_cache_compressed: (nil) 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: persistent_cache: (nil) 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_size: 4096 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_size_deviation: 10 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_restart_interval: 16 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: index_block_restart_interval: 1 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: metadata_block_size: 4096 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: partition_filters: 0 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: use_delta_encoding: 1 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: filter_policy: bloomfilter 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: whole_key_filtering: 1 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: verify_compression: 0 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-04-16T11:44:30.238 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: format_version: 5 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: enable_index_compression: 1 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: block_align: 0 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: max_auto_readahead_size: 262144 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: prepopulate_block_cache: 0 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: initial_auto_readahead_size: 8192 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression: NoCompression 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.num_levels: 7 2024-04-16T11:44:30.239 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T11:44:30.240 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T11:44:30.240 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T11:44:30.241 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T11:44:30.242 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T11:44:30.243 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.inplace_update_support: 0 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.bloom_locality: 0 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.max_successive_merges: 0 2024-04-16T11:44:30.244 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.ttl: 2592000 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.enable_blob_files: false 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.min_blob_size: 0 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T11:44:30.245 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi044/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11823, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a93c19bf-c070-47fa-b590-3fb8fda71725 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267870036197, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267870043967, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 1115603, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11817, "largest_seqno": 12474, "table_properties": {"data_size": 1111707, "index_size": 2088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 7037, "raw_average_key_size": 23, "raw_value_size": 1105066, "raw_average_value_size": 3708, "num_data_blocks": 95, "num_entries": 298, "num_filter_entries": 298, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713267870, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a93c19bf-c070-47fa-b590-3fb8fda71725", "db_session_id": "VMQIEY6SUG9EJ49WGK97", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267870044229, "job": 1, "event": "recovery_finished"} 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-04-16T11:44:30.246 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi044/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c545694000 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: DB pointer 0x55c54567e000 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** DB Stats ** 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.247 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: L0 1/0 1.06 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 250.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: L6 1/0 5.92 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Sum 2/0 6.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 250.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 250.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 250.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T11:44:30.248 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Cumulative compaction: 0.00 GB write, 76.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Interval compaction: 0.00 GB write, 76.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Block cache BinnedLRUCache@0x55c5455a5090#2 capacity: 512.00 MB usage: 3.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,2.33 KB,0.000444055%) Misc(1,0.00 KB,0%) 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T11:44:30.249 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: starting mon.smithi044 rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi044 fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???) e2 preinit fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).mds e13 new map 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).mds e13 print_map 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: e13 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: legacy client fscid: 1 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Filesystem 'cephfs' (1) 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: fs_name cephfs 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: epoch 12 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:44:30.250 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: tableserver 0 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: root 0 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: session_timeout 60 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: session_autoclose 300 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: max_file_size 1099511627776 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: max_xattr_size 65536 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: required_client_features {} 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: last_failure 0 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: last_failure_osd_epoch 0 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: max_mds 1 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: in 0 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: up {0=14456} 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: failed 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: damaged 2024-04-16T11:44:30.251 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: stopped 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: data_pools [3] 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: metadata_pool 2 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: inline_data disabled 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: balancer 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: bal_rank_mask -1 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: standby_count_wanted 1 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: qdb_cluster leader: 0 members: 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: [mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: Standby daemons: 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: [mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: [mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:30.252 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout: [mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).osd e47 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 bash[93001]: 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 2024-04-16T11:44:30.253 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 systemd[1]: Started Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: pgmap v30: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: mon.smithi044 calling monitor election 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: mon.smithi044 is new leader, mons smithi044,smithi045 in quorum (ranks 0,1) 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: monmap epoch 2 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: last_changed 2024-04-16T11:33:49.284289+0000 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: created 2024-04-16T11:30:19.424189+0000 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: min_mon_release 17 (quincy) 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: election_strategy: 1 2024-04-16T11:44:31.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.smithi044 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: mgrmap e30: smithi044.sczobv(active, since 56s), standbys: smithi045.sdlwif 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: overall HEALTH_OK 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: from='mgr.14718 ' entity='' 2024-04-16T11:44:31.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:30 smithi045 ceph-mon[36308]: mgrmap e31: smithi044.sczobv(active, since 56s), standbys: smithi045.sdlwif 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: from='mgr.14718 172.21.15.44:0/147396917' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: pgmap v30: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044 calling monitor election 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mon.smithi044 is new leader, mons smithi044,smithi045 in quorum (ranks 0,1) 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: monmap epoch 2 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:44:31.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: last_changed 2024-04-16T11:33:49.284289+0000 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: created 2024-04-16T11:30:19.424189+0000 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: min_mon_release 17 (quincy) 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: election_strategy: 1 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.smithi044 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mgrmap e30: smithi044.sczobv(active, since 56s), standbys: smithi045.sdlwif 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: overall HEALTH_OK 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: from='mgr.14718 ' entity='' 2024-04-16T11:44:31.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:30 smithi044 ceph-mon[93020]: mgrmap e31: smithi044.sczobv(active, since 56s), standbys: smithi045.sdlwif 2024-04-16T11:44:37.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.154+0000 7f9057800700 1 -- 172.21.15.44:0/2506188990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90500fffd0 msgr2=0x7f90501023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:37.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.154+0000 7f9057800700 1 --2- 172.21.15.44:0/2506188990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90500fffd0 0x7f90501023b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9038004940 tx=0x7f9038009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.155+0000 7f9057800700 1 -- 172.21.15.44:0/2506188990 shutdown_connections 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.155+0000 7f9057800700 1 --2- 172.21.15.44:0/2506188990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f90500fffd0 0x7f90501023b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.155+0000 7f9057800700 1 --2- 172.21.15.44:0/2506188990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9050066290 0x7f90500ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.155+0000 7f9057800700 1 -- 172.21.15.44:0/2506188990 >> 172.21.15.44:0/2506188990 conn(0x7f90500f74c0 msgr2=0x7f90500f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.155+0000 7f9057800700 1 -- 172.21.15.44:0/2506188990 shutdown_connections 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.155+0000 7f9057800700 1 -- 172.21.15.44:0/2506188990 wait complete. 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.156+0000 7f9057800700 1 Processor -- start 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.156+0000 7f9057800700 1 -- start start 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.156+0000 7f9057800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 0x7f905010fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.156+0000 7f9057800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90500fffd0 0x7f9050110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.156+0000 7f9057800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9050110780 con 0x7f9050066290 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.156+0000 7f9057800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9050110fd0 con 0x7f90500fffd0 2024-04-16T11:44:37.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 0x7f905010fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 0x7f905010fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41580/0 (socket says 172.21.15.44:41580) 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 -- 172.21.15.44:0/4254933166 learned_addr learned my addr 172.21.15.44:0/4254933166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90500fffd0 0x7f9050110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90500fffd0 msgr2=0x7f9050110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90500fffd0 0x7f9050110180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 -- 172.21.15.44:0/4254933166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9038005040 con 0x7f9050066290 2024-04-16T11:44:37.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.157+0000 7f9054e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 0x7f905010fc40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9040002a50 tx=0x7f9040004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:37.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.158+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9040007cc0 con 0x7f9050066290 2024-04-16T11:44:37.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.158+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f904000f040 con 0x7f9050066290 2024-04-16T11:44:37.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.158+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90501112b0 con 0x7f9050066290 2024-04-16T11:44:37.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.158+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f904000d940 con 0x7f9050066290 2024-04-16T11:44:37.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.158+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90501117f0 con 0x7f9050066290 2024-04-16T11:44:37.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.159+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905005f5f0 con 0x7f9050066290 2024-04-16T11:44:37.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.160+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 31) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f904000dbd0 con 0x7f9050066290 2024-04-16T11:44:37.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.162+0000 7f904d600700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:37.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.162+0000 7f9047e00700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 msgr2=0x7f90340711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3774957652 2024-04-16T11:44:37.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.162+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f904008e640 con 0x7f9050066290 2024-04-16T11:44:37.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.162+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T11:44:37.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.168+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f904005ae80 con 0x7f9050066290 2024-04-16T11:44:37.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.363+0000 7f9047e00700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 msgr2=0x7f90340711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3774957652 2024-04-16T11:44:37.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.363+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T11:44:37.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.416+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 --> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9050111be0 con 0x7f903406ed20 2024-04-16T11:44:37.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.764+0000 7f9047e00700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 msgr2=0x7f90340711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3774957652 2024-04-16T11:44:37.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:37.764+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T11:44:38.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:38.565+0000 7f9047e00700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 msgr2=0x7f90340711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/3774957652 2024-04-16T11:44:38.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:38.565+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T11:44:40.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:40.058+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 32) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f9040057720 con 0x7f9050066290 2024-04-16T11:44:40.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:40.128+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mgrmap(e 33) v1 ==== 92370+0+0 (secure 0 0 0) 0x7f904008c940 con 0x7f9050066290 2024-04-16T11:44:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:40.129+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 msgr2=0x7f90340711d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T11:44:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:40.129+0000 7f904d600700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:40.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:40 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif restarted 2024-04-16T11:44:40.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:40 smithi045 ceph-mon[36308]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:44:40.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:40 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:44:40.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:40 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:44:40.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:40 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:44:40.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:40 smithi045 ceph-mon[36308]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:44:40.378 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:40 smithi044 ceph-mon[93020]: Standby manager daemon smithi045.sdlwif restarted 2024-04-16T11:44:40.378 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:40 smithi044 ceph-mon[93020]: Standby manager daemon smithi045.sdlwif started 2024-04-16T11:44:40.378 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:40 smithi044 ceph-mon[93020]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/crt"}]: dispatch 2024-04-16T11:44:40.378 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:40 smithi044 ceph-mon[93020]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T11:44:40.379 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:40 smithi044 ceph-mon[93020]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi045.sdlwif/key"}]: dispatch 2024-04-16T11:44:40.379 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:40 smithi044 ceph-mon[93020]: from='mgr.? 172.21.15.45:0/3784428284' entity='mgr.smithi045.sdlwif' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T11:44:41.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.139+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mon.0 v2:172.21.15.44:3300/0 9 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f9040057ff0 con 0x7f9050066290 2024-04-16T11:44:41.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.140+0000 7f904d600700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f90340290d0 0x7f90340294a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.140+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9050111be0 con 0x7f90340290d0 2024-04-16T11:44:41.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.144+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f90340290d0 0x7f90340294a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.147+0000 7f9047e00700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f90340290d0 0x7f90340294a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f903800b3b0 tx=0x7f9038009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:41.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.156+0000 7f904d600700 1 -- 172.21.15.44:0/4254933166 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f9050111be0 con 0x7f90340290d0 2024-04-16T11:44:41.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.160+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f90340290d0 msgr2=0x7f90340294a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.160+0000 7f9057800700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f90340290d0 0x7f90340294a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f903800b3b0 tx=0x7f9038009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:41.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.160+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 msgr2=0x7f905010fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.160+0000 7f9057800700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 0x7f905010fc40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9040002a50 tx=0x7f9040004600 comp rx=0 tx=0).stop 2024-04-16T11:44:41.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 shutdown_connections 2024-04-16T11:44:41.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f90340290d0 0x7f90340294a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:6800/3774957652,v1:172.21.15.44:6801/3774957652] conn(0x7f903406ed20 0x7f90340711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9050066290 0x7f905010fc40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 --2- 172.21.15.44:0/4254933166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90500fffd0 0x7f9050110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 >> 172.21.15.44:0/4254933166 conn(0x7f90500f74c0 msgr2=0x7f90500fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:41.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 shutdown_connections 2024-04-16T11:44:41.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.161+0000 7f9057800700 1 -- 172.21.15.44:0/4254933166 wait complete. 2024-04-16T11:44:41.174 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.313+0000 7f642ba00700 1 -- 172.21.15.44:0/2997031882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6424100320 msgr2=0x7f6424100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.313+0000 7f642ba00700 1 --2- 172.21.15.44:0/2997031882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6424100320 0x7f6424100790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6410004940 tx=0x7f6410009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.313+0000 7f642ba00700 1 -- 172.21.15.44:0/2997031882 shutdown_connections 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.313+0000 7f642ba00700 1 --2- 172.21.15.44:0/2997031882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6424100320 0x7f6424100790 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.313+0000 7f642ba00700 1 --2- 172.21.15.44:0/2997031882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64240ff070 0x7f64240ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.314+0000 7f642ba00700 1 -- 172.21.15.44:0/2997031882 >> 172.21.15.44:0/2997031882 conn(0x7f64240fa950 msgr2=0x7f64240fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.314+0000 7f642ba00700 1 -- 172.21.15.44:0/2997031882 shutdown_connections 2024-04-16T11:44:41.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.314+0000 7f642ba00700 1 -- 172.21.15.44:0/2997031882 wait complete. 2024-04-16T11:44:41.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f642ba00700 1 Processor -- start 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f642ba00700 1 -- start start 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f642ba00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 0x7f642410fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f642ba00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6424100320 0x7f6424110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f642ba00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424110760 con 0x7f64240ff070 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f642ba00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424110fb0 con 0x7f6424100320 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.315+0000 7f6429000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 0x7f642410fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6429000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 0x7f642410fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41696/0 (socket says 172.21.15.44:41696) 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6429000700 1 -- 172.21.15.44:0/1203838188 learned_addr learned my addr 172.21.15.44:0/1203838188 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:41.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6423e00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6424100320 0x7f6424110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6429000700 1 -- 172.21.15.44:0/1203838188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6424100320 msgr2=0x7f6424110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6429000700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6424100320 0x7f6424110160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6429000700 1 -- 172.21.15.44:0/1203838188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410005040 con 0x7f64240ff070 2024-04-16T11:44:41.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.316+0000 7f6429000700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 0x7f642410fc20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f641800c830 tx=0x7f6418004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:41.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.317+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64180106f0 con 0x7f64240ff070 2024-04-16T11:44:41.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.317+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6418010d20 con 0x7f64240ff070 2024-04-16T11:44:41.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.317+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6424111290 con 0x7f64240ff070 2024-04-16T11:44:41.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.317+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64180113a0 con 0x7f64240ff070 2024-04-16T11:44:41.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.317+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64241117d0 con 0x7f64240ff070 2024-04-16T11:44:41.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.318+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642405f5f0 con 0x7f64240ff070 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: mgrmap e32: smithi044.sczobv(active, since 65s), standbys: smithi045.sdlwif 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: Active manager daemon smithi044.sczobv restarted 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: Activating manager daemon smithi044.sczobv 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: mgrmap e33: smithi044.sczobv(active, starting, since 0.0106295s), standbys: smithi045.sdlwif 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:44:41.317 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:44:41.318 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:44:41.319 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:44:41.319 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:41.319 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:41.319 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:44:41.319 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:44:41.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.319+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f6418010850 con 0x7f64240ff070 2024-04-16T11:44:41.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.320+0000 7f6420c00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f640c06e900 0x7f640c070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.320+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f641808e400 con 0x7f64240ff070 2024-04-16T11:44:41.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.321+0000 7f6423e00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f640c06e900 0x7f640c070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.321+0000 7f6423e00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f640c06e900 0x7f640c070db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6410009fd0 tx=0x7f6410009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:41.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.329+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6418057460 con 0x7f64240ff070 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: mgrmap e32: smithi044.sczobv(active, since 65s), standbys: smithi045.sdlwif 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: Active manager daemon smithi044.sczobv restarted 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: Activating manager daemon smithi044.sczobv 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: mgrmap e33: smithi044.sczobv(active, starting, since 0.0106295s), standbys: smithi045.sdlwif 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi044.sczobv", "id": "smithi044.sczobv"}]: dispatch 2024-04-16T11:44:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr metadata", "who": "smithi045.sdlwif", "id": "smithi045.sdlwif"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: Manager daemon smithi044.sczobv is now available 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/mirror_snapshot_schedule"}]: dispatch 2024-04-16T11:44:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:41 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi044.sczobv/trash_purge_schedule"}]: dispatch 2024-04-16T11:44:41.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.592+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6424111bb0 con 0x7f640c06e900 2024-04-16T11:44:41.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.594+0000 7f6420c00700 1 -- 172.21.15.44:0/1203838188 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f6424111bb0 con 0x7f640c06e900 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f640c06e900 msgr2=0x7f640c070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f640c06e900 0x7f640c070db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6410009fd0 tx=0x7f6410009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 msgr2=0x7f642410fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 0x7f642410fc20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f641800c830 tx=0x7f6418004470 comp rx=0 tx=0).stop 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 shutdown_connections 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f640c06e900 0x7f640c070db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f64240ff070 0x7f642410fc20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 --2- 172.21.15.44:0/1203838188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6424100320 0x7f6424110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 >> 172.21.15.44:0/1203838188 conn(0x7f64240fa950 msgr2=0x7f64240fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:41.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 shutdown_connections 2024-04-16T11:44:41.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.597+0000 7f642ba00700 1 -- 172.21.15.44:0/1203838188 wait complete. 2024-04-16T11:44:41.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 -- 172.21.15.44:0/1048023541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 msgr2=0x7fc7440fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 --2- 172.21.15.44:0/1048023541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc7440fd290 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc734004940 tx=0x7fc734009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 -- 172.21.15.44:0/1048023541 shutdown_connections 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 --2- 172.21.15.44:0/1048023541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7440fdff0 0x7fc7440fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 --2- 172.21.15.44:0/1048023541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc7440fd290 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 -- 172.21.15.44:0/1048023541 >> 172.21.15.44:0/1048023541 conn(0x7fc7440f8580 msgr2=0x7fc7440fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.741+0000 7fc749600700 1 -- 172.21.15.44:0/1048023541 shutdown_connections 2024-04-16T11:44:41.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.742+0000 7fc749600700 1 -- 172.21.15.44:0/1048023541 wait complete. 2024-04-16T11:44:41.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.742+0000 7fc749600700 1 Processor -- start 2024-04-16T11:44:41.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.742+0000 7fc749600700 1 -- start start 2024-04-16T11:44:41.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc749600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc744071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc749600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7440fdff0 0x7fc744070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc749600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc744072260 con 0x7fc7440fce80 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc749600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc744070850 con 0x7fc7440fdff0 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc742a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc744071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc742a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc744071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53350/0 (socket says 172.21.15.44:53350) 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc742a00700 1 -- 172.21.15.44:0/3312311920 learned_addr learned my addr 172.21.15.44:0/3312311920 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc73be00700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7440fdff0 0x7fc744070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc742a00700 1 -- 172.21.15.44:0/3312311920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7440fdff0 msgr2=0x7fc744070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc742a00700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7440fdff0 0x7fc744070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:41.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.743+0000 7fc742a00700 1 -- 172.21.15.44:0/3312311920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734005040 con 0x7fc7440fce80 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.744+0000 7fc742a00700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc744071c60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc73400b3b0 tx=0x7fc734009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.744+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73400e040 con 0x7fc7440fce80 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.744+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc734004290 con 0x7fc7440fce80 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.744+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73401c3f0 con 0x7fc7440fce80 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.744+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc744070aa0 con 0x7fc7440fce80 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.744+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc744071060 con 0x7fc7440fce80 2024-04-16T11:44:41.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.746+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74404c710 con 0x7fc7440fce80 2024-04-16T11:44:41.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.746+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 34) v1 ==== 92497+0+0 (secure 0 0 0) 0x7fc73400bbf0 con 0x7fc7440fce80 2024-04-16T11:44:41.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.747+0000 7fc73aa00700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc72806e900 0x7fc728070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:41.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.748+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc73408f660 con 0x7fc7440fce80 2024-04-16T11:44:41.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.748+0000 7fc73be00700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc72806e900 0x7fc728070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:41.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.748+0000 7fc73be00700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc72806e900 0x7fc728070db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc72c004f40 tx=0x7fc72c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:41.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.753+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc734058740 con 0x7fc7440fce80 2024-04-16T11:44:41.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:41.992+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc744102380 con 0x7fc72806e900 2024-04-16T11:44:42.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.004+0000 7fc73aa00700 1 -- 172.21.15.44:0/3312311920 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fc744102380 con 0x7fc72806e900 2024-04-16T11:44:42.002 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:44:42.002 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (109s) 25s ago 12m 14.0M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:44:42.002 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (12m) 25s ago 12m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (11m) 28s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (103s) 25s ago 11m 73.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (5m) 25s ago 5m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (5m) 25s ago 5m 23.3M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (4m) 28s ago 4m 12.6M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (5m) 28s ago 5m 12.6M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (83s) 25s ago 14m 493M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (32s) 28s ago 10m 65.7M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (14m) 25s ago 14m 50.4M 2048M 17.2.7-847-g45317f95 b333581ccdf5 851468b7fc2b 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (10m) 28s ago 10m 37.5M 2048M 17.2.7-847-g45317f95 b333581ccdf5 6de3a87ee8b4 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (117s) 25s ago 12m 7436k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (92s) 28s ago 11m 7792k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:44:42.003 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (9m) 25s ago 9m 46.6M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:44:42.004 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (9m) 25s ago 9m 49.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:44:42.004 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (8m) 25s ago 8m 45.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:44:42.004 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (8m) 28s ago 8m 45.1M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:44:42.004 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (7m) 28s ago 7m 43.8M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:44:42.004 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (6m) 28s ago 6m 45.8M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:44:42.004 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (40s) 25s ago 11m 41.0M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc72806e900 msgr2=0x7fc728070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc72806e900 0x7fc728070db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc72c004f40 tx=0x7fc72c00b040 comp rx=0 tx=0).stop 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 msgr2=0x7fc744071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc744071c60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc73400b3b0 tx=0x7fc734009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 shutdown_connections 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc72806e900 0x7fc728070db0 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc72c004f40 tx=0x7fc72c00b040 comp rx=0 tx=0).stop 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7440fce80 0x7fc744071c60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 --2- 172.21.15.44:0/3312311920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7440fdff0 0x7fc744070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.006+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 >> 172.21.15.44:0/3312311920 conn(0x7fc7440f8580 msgr2=0x7fc7440f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:42.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.007+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 shutdown_connections 2024-04-16T11:44:42.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.007+0000 7fc749600700 1 -- 172.21.15.44:0/3312311920 wait complete. 2024-04-16T11:44:42.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 -- 172.21.15.44:0/429996396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 msgr2=0x7efde4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 --2- 172.21.15.44:0/429996396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 0x7efde4070350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efdd4004970 tx=0x7efdd4009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 -- 172.21.15.44:0/429996396 shutdown_connections 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 --2- 172.21.15.44:0/429996396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde4070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 --2- 172.21.15.44:0/429996396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 0x7efde4070350 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 -- 172.21.15.44:0/429996396 >> 172.21.15.44:0/429996396 conn(0x7efde40f8530 msgr2=0x7efde40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 -- 172.21.15.44:0/429996396 shutdown_connections 2024-04-16T11:44:42.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.162+0000 7efdea600700 1 -- 172.21.15.44:0/429996396 wait complete. 2024-04-16T11:44:42.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.163+0000 7efdea600700 1 Processor -- start 2024-04-16T11:44:42.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.163+0000 7efdea600700 1 -- start start 2024-04-16T11:44:42.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efdea600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde410fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:42.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efdea600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 0x7efde41101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:42.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efdea600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4110870 con 0x7efde4070890 2024-04-16T11:44:42.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efdea600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4111010 con 0x7efde4071e40 2024-04-16T11:44:42.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efde3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde410fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:42.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efde3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde410fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53366/0 (socket says 172.21.15.44:53366) 2024-04-16T11:44:42.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.164+0000 7efde3400700 1 -- 172.21.15.44:0/3407364586 learned_addr learned my addr 172.21.15.44:0/3407364586 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:42.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efde3400700 1 -- 172.21.15.44:0/3407364586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 msgr2=0x7efde41101e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efde3400700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 0x7efde41101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efde3400700 1 -- 172.21.15.44:0/3407364586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd4005040 con 0x7efde4070890 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efde3400700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde410fca0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efdd4000c00 tx=0x7efdd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdd400e040 con 0x7efde4070890 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdd400bce0 con 0x7efde4070890 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde4111290 con 0x7efde4070890 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdd401c6a0 con 0x7efde4070890 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.165+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde41117d0 con 0x7efde4070890 2024-04-16T11:44:42.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.167+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde405f5f0 con 0x7efde4070890 2024-04-16T11:44:42.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.169+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7efdd4013b10 con 0x7efde4070890 2024-04-16T11:44:42.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.169+0000 7efddb400700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efdc806eca0 0x7efdc8071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:42.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.169+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efdd4090830 con 0x7efde4070890 2024-04-16T11:44:42.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.170+0000 7efde2a00700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efdc806eca0 0x7efdc8071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:42.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.171+0000 7efde2a00700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efdc806eca0 0x7efdc8071150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efdd0006fd0 tx=0x7efdd0006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:42.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.174+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efdd405d120 con 0x7efde4070890 2024-04-16T11:44:42.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: [16/Apr/2024:11:44:41] ENGINE Bus STARTING 2024-04-16T11:44:42.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: mgrmap e34: smithi044.sczobv(active, since 1.02795s), standbys: smithi045.sdlwif 2024-04-16T11:44:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: pgmap v3: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: [16/Apr/2024:11:44:41] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:44:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: [16/Apr/2024:11:44:41] ENGINE Serving on http://172.21.15.44:8765 2024-04-16T11:44:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: [16/Apr/2024:11:44:41] ENGINE Bus STARTED 2024-04-16T11:44:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:42 smithi044 ceph-mon[93020]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:42.261 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: [16/Apr/2024:11:44:41] ENGINE Bus STARTING 2024-04-16T11:44:42.261 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: mgrmap e34: smithi044.sczobv(active, since 1.02795s), standbys: smithi045.sdlwif 2024-04-16T11:44:42.261 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:42.261 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: pgmap v3: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:42.262 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: [16/Apr/2024:11:44:41] ENGINE Serving on https://172.21.15.44:7150 2024-04-16T11:44:42.262 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: [16/Apr/2024:11:44:41] ENGINE Serving on http://172.21.15.44:8765 2024-04-16T11:44:42.262 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: [16/Apr/2024:11:44:41] ENGINE Bus STARTED 2024-04-16T11:44:42.262 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:42 smithi045 ceph-mon[36308]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.511+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efde4107b70 con 0x7efde4070890 2024-04-16T11:44:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.512+0000 7efddb400700 1 -- 172.21.15.44:0/3407364586 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7efdd405ccb0 con 0x7efde4070890 2024-04-16T11:44:42.510 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:44:42.510 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:44:42.510 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 11, 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T11:44:42.511 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:44:42.512 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efdc806eca0 msgr2=0x7efdc8071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efdc806eca0 0x7efdc8071150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efdd0006fd0 tx=0x7efdd0006db0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 msgr2=0x7efde410fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde410fca0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efdd4000c00 tx=0x7efdd4009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 shutdown_connections 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efdc806eca0 0x7efdc8071150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde4070890 0x7efde410fca0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 --2- 172.21.15.44:0/3407364586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde4071e40 0x7efde41101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 >> 172.21.15.44:0/3407364586 conn(0x7efde40f8530 msgr2=0x7efde40f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 shutdown_connections 2024-04-16T11:44:42.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.515+0000 7efdea600700 1 -- 172.21.15.44:0/3407364586 wait complete. 2024-04-16T11:44:42.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 -- 172.21.15.44:0/3555121511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8104520 msgr2=0x7fbdc8104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3555121511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8104520 0x7fbdc8104990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004940 tx=0x7fbdb0009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:42.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 -- 172.21.15.44:0/3555121511 shutdown_connections 2024-04-16T11:44:42.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3555121511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8104520 0x7fbdc8104990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3555121511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdc8103270 0x7fbdc8103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 -- 172.21.15.44:0/3555121511 >> 172.21.15.44:0/3555121511 conn(0x7fbdc8072530 msgr2=0x7fbdc8074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:42.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 -- 172.21.15.44:0/3555121511 shutdown_connections 2024-04-16T11:44:42.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.681+0000 7fbdcca00700 1 -- 172.21.15.44:0/3555121511 wait complete. 2024-04-16T11:44:42.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.682+0000 7fbdcca00700 1 Processor -- start 2024-04-16T11:44:42.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.682+0000 7fbdcca00700 1 -- start start 2024-04-16T11:44:42.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.682+0000 7fbdcca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 0x7fbdc8113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.682+0000 7fbdcca00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdc8104520 0x7fbdc8114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.682+0000 7fbdcca00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8114a60 con 0x7fbdc8103270 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.682+0000 7fbdcca00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc81151a0 con 0x7fbdc8104520 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 0x7fbdc8113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 0x7fbdc8113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53404/0 (socket says 172.21.15.44:53404) 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 -- 172.21.15.44:0/3210128139 learned_addr learned my addr 172.21.15.44:0/3210128139 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdbea00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdc8104520 0x7fbdc8114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 -- 172.21.15.44:0/3210128139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdc8104520 msgr2=0x7fbdc8114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdc8104520 0x7fbdc8114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 -- 172.21.15.44:0/3210128139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0005040 con 0x7fbdc8103270 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.683+0000 7fbdc7400700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 0x7fbdc8113f20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8001d00 tx=0x7fbdb8009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.684+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb8004020 con 0x7fbdc8103270 2024-04-16T11:44:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.684+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdb800f040 con 0x7fbdc8103270 2024-04-16T11:44:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.684+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb8009310 con 0x7fbdc8103270 2024-04-16T11:44:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.684+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc8115480 con 0x7fbdc8103270 2024-04-16T11:44:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.684+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8115a40 con 0x7fbdc8103270 2024-04-16T11:44:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.685+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc80760a0 con 0x7fbdc8103270 2024-04-16T11:44:42.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.686+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7fbdb8009490 con 0x7fbdc8103270 2024-04-16T11:44:42.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.687+0000 7fbdbfe00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbdac06ec90 0x7fbdac071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:42.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.687+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbdb808e3a0 con 0x7fbdc8103270 2024-04-16T11:44:42.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.687+0000 7fbdbea00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbdac06ec90 0x7fbdac071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.688+0000 7fbdbea00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbdac06ec90 0x7fbdac071140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004f90 tx=0x7fbdb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:42.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.692+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbdb805ab50 con 0x7fbdc8103270 2024-04-16T11:44:42.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.982+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbdc804bed0 con 0x7fbdc8103270 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.982+0000 7fbdbfe00700 1 -- 172.21.15.44:0/3210128139 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7fbdb805a6e0 con 0x7fbdc8103270 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:44:42.981 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:44:42.982 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:42.983 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:44:42.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbdac06ec90 msgr2=0x7fbdac071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbdac06ec90 0x7fbdac071140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004f90 tx=0x7fbdb0009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:42.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 msgr2=0x7fbdc8113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:42.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 0x7fbdc8113f20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8001d00 tx=0x7fbdb8009f30 comp rx=0 tx=0).stop 2024-04-16T11:44:42.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 shutdown_connections 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbdac06ec90 0x7fbdac071140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdc8103270 0x7fbdc8113f20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 --2- 172.21.15.44:0/3210128139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdc8104520 0x7fbdc8114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.985+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 >> 172.21.15.44:0/3210128139 conn(0x7fbdc8072530 msgr2=0x7fbdc8074ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.986+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 shutdown_connections 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:42.986+0000 7fbdcca00700 1 -- 172.21.15.44:0/3210128139 wait complete. 2024-04-16T11:44:42.985 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:44:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 -- 172.21.15.44:0/3087992523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 msgr2=0x7fd164070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 --2- 172.21.15.44:0/3087992523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd164070ce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd150004940 tx=0x7fd150009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:43.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 -- 172.21.15.44:0/3087992523 shutdown_connections 2024-04-16T11:44:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 --2- 172.21.15.44:0/3087992523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd164070ce0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 --2- 172.21.15.44:0/3087992523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd164071e20 0x7fd164070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 -- 172.21.15.44:0/3087992523 >> 172.21.15.44:0/3087992523 conn(0x7fd1640f83a0 msgr2=0x7fd1640fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.151+0000 7fd16ae00700 1 -- 172.21.15.44:0/3087992523 shutdown_connections 2024-04-16T11:44:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.152+0000 7fd16ae00700 1 -- 172.21.15.44:0/3087992523 wait complete. 2024-04-16T11:44:43.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.152+0000 7fd16ae00700 1 Processor -- start 2024-04-16T11:44:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.153+0000 7fd16ae00700 1 -- start start 2024-04-16T11:44:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.153+0000 7fd16ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd16410f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.153+0000 7fd16ae00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd164071e20 0x7fd16410ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.153+0000 7fd16ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd164110500 con 0x7fd164070870 2024-04-16T11:44:43.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.153+0000 7fd16ae00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd164110d50 con 0x7fd164071e20 2024-04-16T11:44:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.153+0000 7fd169a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd16410f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd16410f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53412/0 (socket says 172.21.15.44:53412) 2024-04-16T11:44:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169a00700 1 -- 172.21.15.44:0/1500648786 learned_addr learned my addr 172.21.15.44:0/1500648786 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169000700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd164071e20 0x7fd16410ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169a00700 1 -- 172.21.15.44:0/1500648786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd164071e20 msgr2=0x7fd16410ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169a00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd164071e20 0x7fd16410ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169a00700 1 -- 172.21.15.44:0/1500648786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150005040 con 0x7fd164070870 2024-04-16T11:44:43.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.154+0000 7fd169a00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd16410f9c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd15800eb70 tx=0x7fd15800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:43.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.156+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd158007cc0 con 0x7fd164070870 2024-04-16T11:44:43.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.156+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd15800f040 con 0x7fd164070870 2024-04-16T11:44:43.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.156+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164110fd0 con 0x7fd164070870 2024-04-16T11:44:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.156+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd15800cb20 con 0x7fd164070870 2024-04-16T11:44:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.157+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd164111510 con 0x7fd164070870 2024-04-16T11:44:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.158+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1640fc020 con 0x7fd164070870 2024-04-16T11:44:43.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.159+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7fd15800cc80 con 0x7fd164070870 2024-04-16T11:44:43.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.160+0000 7fd161600700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd14c06ec90 0x7fd14c071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:43.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.160+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd15808e3d0 con 0x7fd164070870 2024-04-16T11:44:43.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.161+0000 7fd169000700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd14c06ec90 0x7fd14c071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:43.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.162+0000 7fd169000700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd14c06ec90 0x7fd14c071140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd150009fd0 tx=0x7fd150009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:43.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.165+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd15805ac30 con 0x7fd164070870 2024-04-16T11:44:43.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.435+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd164111950 con 0x7fd14c06ec90 2024-04-16T11:44:43.434 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:43 smithi044 ceph-mon[93020]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:43.434 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:43 smithi044 ceph-mon[93020]: pgmap v4: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:43.434 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:43 smithi044 ceph-mon[93020]: mgrmap e35: smithi044.sczobv(active, since 2s), standbys: smithi045.sdlwif 2024-04-16T11:44:43.434 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:43 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3407364586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:43.434 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:43 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3210128139' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:44:43.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.436+0000 7fd161600700 1 -- 172.21.15.44:0/1500648786 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fd164111950 con 0x7fd14c06ec90 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:44:43.435 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:44:43.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.439+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd14c06ec90 msgr2=0x7fd14c071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.439+0000 7fd16ae00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd14c06ec90 0x7fd14c071140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd150009fd0 tx=0x7fd150009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:43.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.439+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 msgr2=0x7fd16410f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.439+0000 7fd16ae00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd16410f9c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd15800eb70 tx=0x7fd15800b040 comp rx=0 tx=0).stop 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 shutdown_connections 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd14c06ec90 0x7fd14c071140 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd164070870 0x7fd16410f9c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 --2- 172.21.15.44:0/1500648786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd164071e20 0x7fd16410ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 >> 172.21.15.44:0/1500648786 conn(0x7fd1640f83a0 msgr2=0x7fd1640f9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 shutdown_connections 2024-04-16T11:44:43.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.440+0000 7fd16ae00700 1 -- 172.21.15.44:0/1500648786 wait complete. 2024-04-16T11:44:43.520 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:43 smithi045 ceph-mon[36308]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:43.520 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:43 smithi045 ceph-mon[36308]: pgmap v4: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:43.520 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:43 smithi045 ceph-mon[36308]: mgrmap e35: smithi044.sczobv(active, since 2s), standbys: smithi045.sdlwif 2024-04-16T11:44:43.520 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:43 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3407364586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:43.520 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:43 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/3210128139' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:44:43.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 -- 172.21.15.44:0/3381026600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 msgr2=0x7fe5d80ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/3381026600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d80ff410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8004940 tx=0x7fe5c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:43.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 -- 172.21.15.44:0/3381026600 shutdown_connections 2024-04-16T11:44:43.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/3381026600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d81002b0 0x7fe5d8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/3381026600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d80ff410 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 -- 172.21.15.44:0/3381026600 >> 172.21.15.44:0/3381026600 conn(0x7fe5d80fa790 msgr2=0x7fe5d80fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:43.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 -- 172.21.15.44:0/3381026600 shutdown_connections 2024-04-16T11:44:43.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.592+0000 7fe5ddc00700 1 -- 172.21.15.44:0/3381026600 wait complete. 2024-04-16T11:44:43.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.593+0000 7fe5ddc00700 1 Processor -- start 2024-04-16T11:44:43.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.593+0000 7fe5ddc00700 1 -- start start 2024-04-16T11:44:43.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5ddc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d810fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:43.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5ddc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d81002b0 0x7fe5d8110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:43.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5ddc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8110700 con 0x7fe5d80ff000 2024-04-16T11:44:43.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5ddc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d81150a0 con 0x7fe5d81002b0 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d810fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d810fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53432/0 (socket says 172.21.15.44:53432) 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6a00700 1 -- 172.21.15.44:0/2670253127 learned_addr learned my addr 172.21.15.44:0/2670253127 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6000700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d81002b0 0x7fe5d8110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6a00700 1 -- 172.21.15.44:0/2670253127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d81002b0 msgr2=0x7fe5d8110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6a00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d81002b0 0x7fe5d8110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.594+0000 7fe5d6a00700 1 -- 172.21.15.44:0/2670253127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8005040 con 0x7fe5d80ff000 2024-04-16T11:44:43.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.595+0000 7fe5d6a00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d810fb30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009150 tx=0x7fe5c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:43.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.595+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5c800e040 con 0x7fe5d80ff000 2024-04-16T11:44:43.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.595+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5c800bbf0 con 0x7fe5d80ff000 2024-04-16T11:44:43.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.595+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5c801c630 con 0x7fe5d80ff000 2024-04-16T11:44:43.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.595+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d81152a0 con 0x7fe5d80ff000 2024-04-16T11:44:43.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.595+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d8115810 con 0x7fe5d80ff000 2024-04-16T11:44:43.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.596+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d804c710 con 0x7fe5d80ff000 2024-04-16T11:44:43.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.598+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 35) v1 ==== 92791+0+0 (secure 0 0 0) 0x7fe5c800bd60 con 0x7fe5d80ff000 2024-04-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.599+0000 7fe5cea00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5bc06ec90 0x7fe5bc071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.599+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe5c808fd70 con 0x7fe5d80ff000 2024-04-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.599+0000 7fe5d6000700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5bc06ec90 0x7fe5bc071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.600+0000 7fe5d6000700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5bc06ec90 0x7fe5bc071140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4006fd0 tx=0x7fe5c4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:44:43.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.604+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe5c805c5a0 con 0x7fe5d80ff000 2024-04-16T11:44:43.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.933+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe5d8115c80 con 0x7fe5d80ff000 2024-04-16T11:44:43.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.934+0000 7fe5cea00700 1 -- 172.21.15.44:0/2670253127 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fe5c805c130 con 0x7fe5d80ff000 2024-04-16T11:44:43.932 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5bc06ec90 msgr2=0x7fe5bc071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5bc06ec90 0x7fe5bc071140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4006fd0 tx=0x7fe5c4008040 comp rx=0 tx=0).stop 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 msgr2=0x7fe5d810fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d810fb30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009150 tx=0x7fe5c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 shutdown_connections 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5bc06ec90 0x7fe5bc071140 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d80ff000 0x7fe5d810fb30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 --2- 172.21.15.44:0/2670253127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d81002b0 0x7fe5d8110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:44:43.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 >> 172.21.15.44:0/2670253127 conn(0x7fe5d80fa790 msgr2=0x7fe5d80fbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:44:43.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 shutdown_connections 2024-04-16T11:44:43.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:44:43.937+0000 7fe5ddc00700 1 -- 172.21.15.44:0/2670253127 wait complete. 2024-04-16T11:44:44.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:44 smithi044 ceph-mon[93020]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:44.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:44 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2670253127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:44:44.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:44 smithi045 ceph-mon[36308]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:44:44.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:44 smithi045 ceph-mon[36308]: from='client.? 172.21.15.44:0/2670253127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:44:45.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:45 smithi044 ceph-mon[93020]: pgmap v5: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:45 smithi044 ceph-mon[93020]: mgrmap e36: smithi044.sczobv(active, since 4s), standbys: smithi045.sdlwif 2024-04-16T11:44:45.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:45 smithi045 ceph-mon[36308]: pgmap v5: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:45.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:45 smithi045 ceph-mon[36308]: mgrmap e36: smithi044.sczobv(active, since 4s), standbys: smithi045.sdlwif 2024-04-16T11:44:46.696 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:46.696 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:46.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:46 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:46.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:46 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:47.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:47 smithi044 ceph-mon[93020]: pgmap v6: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:47.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:47 smithi045 ceph-mon[36308]: pgmap v6: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:44:48.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:48 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:48 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:48 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:48 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:48.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:49.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:49 smithi045 ceph-mon[36308]: pgmap v7: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T11:44:49.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:49 smithi044 ceph-mon[93020]: pgmap v7: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T11:44:50.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:50 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:50.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:50 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:51.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:51.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 180, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77539 2024-04-16T11:44:51.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 180, success, outsize: 120 2024-04-16T11:44:51.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 182, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77627 2024-04-16T11:44:51.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 182, success, outsize: 120 2024-04-16T11:44:51.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 184, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77718 2024-04-16T11:44:51.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 184, success, outsize: 120 2024-04-16T11:44:51.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 186, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77815 2024-04-16T11:44:51.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 186, success, outsize: 120 2024-04-16T11:44:51.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 188, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77909 2024-04-16T11:44:51.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 188, success, outsize: 120 2024-04-16T11:44:51.830 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 190, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77997 2024-04-16T11:44:51.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 190, success, outsize: 120 2024-04-16T11:44:51.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78088 2024-04-16T11:44:51.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 192, success, outsize: 120 2024-04-16T11:44:51.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78185 2024-04-16T11:44:51.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 194, success, outsize: 120 2024-04-16T11:44:51.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78279 2024-04-16T11:44:51.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 196, success, outsize: 120 2024-04-16T11:44:51.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 198, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78367 2024-04-16T11:44:51.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 198, success, outsize: 120 2024-04-16T11:44:51.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 200, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78458 2024-04-16T11:44:51.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 200, success, outsize: 120 2024-04-16T11:44:51.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 202, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78555 2024-04-16T11:44:51.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 202, success, outsize: 120 2024-04-16T11:44:51.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 204, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78650 2024-04-16T11:44:51.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 204, success, outsize: 120 2024-04-16T11:44:51.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 206, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78739 2024-04-16T11:44:51.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 206, success, outsize: 120 2024-04-16T11:44:51.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 208, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78831 2024-04-16T11:44:51.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 208, success, outsize: 120 2024-04-16T11:44:52.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 210, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78928 2024-04-16T11:44:52.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 210, success, outsize: 120 2024-04-16T11:44:52.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 212, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79022 2024-04-16T11:44:52.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 212, success, outsize: 120 2024-04-16T11:44:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 214, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79110 2024-04-16T11:44:52.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 214, success, outsize: 120 2024-04-16T11:44:52.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 216, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79201 2024-04-16T11:44:52.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 216, success, outsize: 120 2024-04-16T11:44:52.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:51 smithi045 ceph-mon[36308]: pgmap v8: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T11:44:52.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 218, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79298 2024-04-16T11:44:52.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 218, success, outsize: 120 2024-04-16T11:44:52.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 220, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79392 2024-04-16T11:44:52.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 220, success, outsize: 120 2024-04-16T11:44:52.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 222, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79481 2024-04-16T11:44:52.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 222, success, outsize: 120 2024-04-16T11:44:52.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 224, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79573 2024-04-16T11:44:52.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 224, success, outsize: 120 2024-04-16T11:44:52.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 226, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79673 2024-04-16T11:44:52.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 226, success, outsize: 120 2024-04-16T11:44:52.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 228, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79767 2024-04-16T11:44:52.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 228, success, outsize: 120 2024-04-16T11:44:52.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 230, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79855 2024-04-16T11:44:52.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 230, success, outsize: 120 2024-04-16T11:44:52.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 232, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79946 2024-04-16T11:44:52.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 232, success, outsize: 120 2024-04-16T11:44:52.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 234, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 80043 2024-04-16T11:44:52.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 234, success, outsize: 120 2024-04-16T11:44:52.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:51 smithi044 ceph-mon[93020]: pgmap v8: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T11:44:54.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:53 smithi045 ceph-mon[36308]: pgmap v9: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T11:44:54.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:53 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:54.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:53 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:54.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:53 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:54.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:53 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:44:54.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:53 smithi044 ceph-mon[93020]: pgmap v9: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T11:44:54.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:44:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:44:55.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:54 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:44:55.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:54 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:44:55.176 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:54 smithi044 ceph-mon[93020]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-16T11:44:55.176 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:54 smithi044 ceph-mon[93020]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-16T11:44:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:55 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:44:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:55 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:44:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:55 smithi045 ceph-mon[36308]: pgmap v10: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:44:56.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:55 smithi045 ceph-mon[36308]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:44:56.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:55 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:56.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:55 smithi044 ceph-mon[93020]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:44:56.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:55 smithi044 ceph-mon[93020]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T11:44:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:55 smithi044 ceph-mon[93020]: pgmap v10: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:44:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:55 smithi044 ceph-mon[93020]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:44:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:44:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:44:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:44:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:44:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:56 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: Updating smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: Updating smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:44:58.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:57 smithi045 ceph-mon[36308]: pgmap v11: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:44:58.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:57 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:58.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:57 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:58.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:57 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T11:44:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:57 smithi044 ceph-mon[93020]: pgmap v11: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:44:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:44:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:44:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T11:45:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:59 smithi045 ceph-mon[36308]: pgmap v12: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:45:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:59 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:59 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:45:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:59 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:45:00.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:44:59 smithi045 ceph-mon[36308]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:59 smithi044 ceph-mon[93020]: pgmap v12: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T11:45:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:45:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:45:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:44:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:00 smithi045 ceph-mon[36308]: Upgrade: Updating mon.smithi045 2024-04-16T11:45:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:00 smithi045 ceph-mon[36308]: Deploying daemon mon.smithi045 on smithi045 2024-04-16T11:45:01.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:00 smithi044 ceph-mon[93020]: Upgrade: Updating mon.smithi045 2024-04-16T11:45:01.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:00 smithi044 ceph-mon[93020]: Deploying daemon mon.smithi045 on smithi045 2024-04-16T11:45:02.022 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:01 smithi045 ceph-mon[36308]: pgmap v13: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:01 smithi044 ceph-mon[93020]: pgmap v13: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:02 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:45:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:02 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045[36296]: 2024-04-16T11:45:02.493+0000 7f334f000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:45:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:02 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045[36296]: 2024-04-16T11:45:02.493+0000 7f334f000700 -1 mon.smithi045@1(peon) e2 *** Got Signal Terminated *** 2024-04-16T11:45:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:02 smithi045 podman[81261]: 2024-04-16 11:45:02.669480146 +0000 UTC m=+0.306204157 container died 6de3a87ee8b4e53adbf80683e316e293e1d5b85f3fe3d92c49367827fb22ef13 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-16T11:45:03.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:03 smithi045 podman[81261]: 2024-04-16 11:45:03.211983177 +0000 UTC m=+0.848707196 container cleanup 6de3a87ee8b4e53adbf80683e316e293e1d5b85f3fe3d92c49367827fb22ef13 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T11:45:03.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:03 smithi045 bash[81261]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045 2024-04-16T11:45:03.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:03 smithi045 podman[81274]: 2024-04-16 11:45:03.48800594 +0000 UTC m=+0.796132725 container remove 6de3a87ee8b4e53adbf80683e316e293e1d5b85f3fe3d92c49367827fb22ef13 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:45:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:04 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi045.service: Deactivated successfully. 2024-04-16T11:45:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:04 smithi045 systemd[1]: Stopped Ceph mon.smithi045 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:45:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:04 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi045.service: Consumed 14.878s CPU time. 2024-04-16T11:45:04.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:04 smithi045 systemd[1]: Starting Ceph mon.smithi045 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:45:05.069 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:04 smithi045 podman[81404]: 2024-04-16 11:45:04.682266261 +0000 UTC m=+0.040770615 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:45:05.069 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:04 smithi045 podman[81404]: 2024-04-16 11:45:04.814635145 +0000 UTC m=+0.173139526 container create 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 podman[81404]: 2024-04-16 11:45:05.071797027 +0000 UTC m=+0.430301397 container init 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 podman[81404]: 2024-04-16 11:45:05.082763755 +0000 UTC m=+0.441268143 container start 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: pidfile_write: ignore empty --pid-file 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: load: jerasure load: lrc 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: RocksDB version: 7.9.2 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Git sha 0 2024-04-16T11:45:05.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: DB SUMMARY 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: DB Session ID: RXE6B4JR83TL80B9CI29 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: CURRENT file: CURRENT 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1072 Bytes 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi045/store.db dir, Total Num: 1, files: 000027.sst 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi045/store.db: 000025.log size: 4761792 ; 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.error_if_exists: 0 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.create_if_missing: 0 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.paranoid_checks: 1 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T11:45:05.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T11:45:05.148 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.env: 0x55a2a3b22cc0 2024-04-16T11:45:05.148 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T11:45:05.148 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.info_log: 0x55a2e411bae0 2024-04-16T11:45:05.148 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.statistics: (nil) 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.use_fsync: 0 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_log_file_size: 0 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.allow_fallocate: 1 2024-04-16T11:45:05.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.use_direct_reads: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.db_log_dir: 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.wal_dir: 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T11:45:05.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.write_buffer_manager: 0x55a2e40ec1e0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.unordered_write: 0 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T11:45:05.151 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.row_cache: None 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.wal_filter: None 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.two_write_queues: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.wal_compression: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.atomic_flush: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.log_readahead_size: 0 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T11:45:05.152 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_background_jobs: 2 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_background_compactions: -1 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_subcompactions: 1 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T11:45:05.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_open_files: -1 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_background_flushes: -1 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Compression algorithms supported: 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kZSTD supported: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kXpressCompression supported: 0 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kZlibCompression supported: 1 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kSnappyCompression supported: 1 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kLZ4Compression supported: 1 2024-04-16T11:45:05.154 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: kBZip2Compression supported: 0 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000009 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.merge_operator: 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_filter: None 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a2e411bcc0) 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T11:45:05.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_type: 0 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_block_index_type: 0 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_shortening: 1 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: checksum: 4 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: no_block_cache: 0 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache: 0x55a2e411f090 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_options: 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: capacity : 536870912 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: num_shard_bits : 4 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: strict_capacity_limit : 0 2024-04-16T11:45:05.156 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_compressed: (nil) 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: persistent_cache: (nil) 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_size: 4096 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_size_deviation: 10 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_restart_interval: 16 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_block_restart_interval: 1 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: metadata_block_size: 4096 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: partition_filters: 0 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: use_delta_encoding: 1 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: filter_policy: bloomfilter 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: whole_key_filtering: 1 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: verify_compression: 0 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-04-16T11:45:05.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: format_version: 5 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: enable_index_compression: 1 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_align: 0 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: max_auto_readahead_size: 262144 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: prepopulate_block_cache: 0 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: initial_auto_readahead_size: 8192 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T11:45:05.158 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression: NoCompression 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.num_levels: 7 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T11:45:05.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T11:45:05.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T11:45:05.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.inplace_update_support: 0 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.bloom_locality: 0 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.max_successive_merges: 0 2024-04-16T11:45:05.162 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.ttl: 2592000 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.enable_blob_files: false 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.min_blob_size: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T11:45:05.163 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 11903, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1a18e211-a6e0-448c-968d-208e747f4eda 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267905140328, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-16T11:45:05.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-16T11:45:05.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267905159176, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2965462, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11900, "largest_seqno": 13194, "table_properties": {"data_size": 2959318, "index_size": 3691, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1413, "raw_key_size": 12626, "raw_average_key_size": 23, "raw_value_size": 2947565, "raw_average_value_size": 5458, "num_data_blocks": 169, "num_entries": 540, "num_filter_entries": 540, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713267905, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1a18e211-a6e0-448c-968d-208e747f4eda", "db_session_id": "RXE6B4JR83TL80B9CI29", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-16T11:45:05.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713267905159502, "job": 1, "event": "recovery_finished"} 2024-04-16T11:45:05.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi045/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a2e420e000 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: DB pointer 0x55a2e41f8000 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** DB Stats ** 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T11:45:05.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: L0 1/0 2.83 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 486.8 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: L6 1/0 5.92 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T11:45:05.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Sum 2/0 8.75 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 486.8 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 486.8 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 486.8 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T11:45:05.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative compaction: 0.00 GB write, 114.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval compaction: 0.00 GB write, 114.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Block cache BinnedLRUCache@0x55a2e411f090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: starting mon.smithi045 rank 1 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi045 fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???) e2 preinit fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).mds e13 new map 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).mds e13 print_map 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: e13 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: legacy client fscid: 1 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Filesystem 'cephfs' (1) 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: fs_name cephfs 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: epoch 12 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: tableserver 0 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: root 0 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: session_timeout 60 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: session_autoclose 300 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: max_file_size 1099511627776 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: max_xattr_size 65536 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: required_client_features {} 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: last_failure 0 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: last_failure_osd_epoch 0 2024-04-16T11:45:05.336 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: max_mds 1 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: in 0 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: up {0=14456} 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: failed 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: damaged 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: stopped 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_pools [3] 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: metadata_pool 2 2024-04-16T11:45:05.337 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: inline_data disabled 2024-04-16T11:45:05.338 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: balancer 2024-04-16T11:45:05.338 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: bal_rank_mask -1 2024-04-16T11:45:05.338 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: standby_count_wanted 1 2024-04-16T11:45:05.338 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: qdb_cluster leader: 0 members: 2024-04-16T11:45:05.338 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: [mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:05.338 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.339 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.339 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Standby daemons: 2024-04-16T11:45:05.339 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-04-16T11:45:05.339 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: [mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:05.339 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: [mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:05.339 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: [mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 ceph-mon[81423]: mon.smithi045@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 bash[81404]: 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c 2024-04-16T11:45:05.340 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:05 smithi045 systemd[1]: Started Ceph mon.smithi045 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:45:06.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: mon.smithi045 calling monitor election 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: mon.smithi044 calling monitor election 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: mon.smithi044 is new leader, mons smithi044,smithi045 in quorum (ranks 0,1) 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: monmap epoch 3 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: last_changed 2024-04-16T11:45:05.709806+0000 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: created 2024-04-16T11:30:19.424189+0000 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: min_mon_release 19 (squid) 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: election_strategy: 1 2024-04-16T11:45:06.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.smithi044 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: mgrmap e36: smithi044.sczobv(active, since 25s), standbys: smithi045.sdlwif 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: overall HEALTH_OK 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:06.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:06 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi044"}]: dispatch 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: mon.smithi045 calling monitor election 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: mon.smithi044 calling monitor election 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: mon.smithi044 is new leader, mons smithi044,smithi045 in quorum (ranks 0,1) 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: monmap epoch 3 2024-04-16T11:45:07.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: last_changed 2024-04-16T11:45:05.709806+0000 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: created 2024-04-16T11:30:19.424189+0000 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: min_mon_release 19 (squid) 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: election_strategy: 1 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.smithi044 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: mgrmap e36: smithi044.sczobv(active, since 25s), standbys: smithi045.sdlwif 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: overall HEALTH_OK 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:07.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:06 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:45:08.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:07 smithi045 ceph-mon[81423]: pgmap v16: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:07 smithi044 ceph-mon[93020]: pgmap v16: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:10.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:09 smithi045 ceph-mon[81423]: pgmap v17: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:10.173 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:09 smithi044 ceph-mon[93020]: pgmap v17: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:11 smithi044 ceph-mon[93020]: pgmap v18: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:11.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:11 smithi045 ceph-mon[81423]: pgmap v18: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:11.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:11.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:11.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:13.776 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:13 smithi045 ceph-mon[81423]: pgmap v19: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:13.776 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:13 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:13.776 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:13 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:13 smithi044 ceph-mon[93020]: pgmap v19: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:13 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:13.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:13 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 -- 172.21.15.44:0/2965250610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4104520 msgr2=0x7fa0b4104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/2965250610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4104520 0x7fa0b4104990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa09c004940 tx=0x7fa09c009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 -- 172.21.15.44:0/2965250610 shutdown_connections 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/2965250610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4104520 0x7fa0b4104990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/2965250610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4103270 0x7fa0b4103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 -- 172.21.15.44:0/2965250610 >> 172.21.15.44:0/2965250610 conn(0x7fa0b4072530 msgr2=0x7fa0b4074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 -- 172.21.15.44:0/2965250610 shutdown_connections 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.111+0000 7fa0b3e00700 1 -- 172.21.15.44:0/2965250610 wait complete. 2024-04-16T11:45:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.112+0000 7fa0b3e00700 1 Processor -- start 2024-04-16T11:45:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.112+0000 7fa0b3e00700 1 -- start start 2024-04-16T11:45:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.112+0000 7fa0b3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 0x7fa0b4113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.112+0000 7fa0b3e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4104520 0x7fa0b4114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.112+0000 7fa0b3e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4114a60 con 0x7fa0b4103270 2024-04-16T11:45:14.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.112+0000 7fa0b3e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41151a0 con 0x7fa0b4104520 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 0x7fa0b4113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0b2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 0x7fa0b4113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47756/0 (socket says 172.21.15.44:47756) 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0b2a00700 1 -- 172.21.15.44:0/4328456 learned_addr learned my addr 172.21.15.44:0/4328456 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0abe00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4104520 0x7fa0b4114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0b2a00700 1 -- 172.21.15.44:0/4328456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4104520 msgr2=0x7fa0b4114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0b2a00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4104520 0x7fa0b4114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.113+0000 7fa0b2a00700 1 -- 172.21.15.44:0/4328456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c005040 con 0x7fa0b4103270 2024-04-16T11:45:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.114+0000 7fa0b2a00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 0x7fa0b4113f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c810 tx=0x7fa0a4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.114+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a4012810 con 0x7fa0b4103270 2024-04-16T11:45:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.114+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0a4012e40 con 0x7fa0b4103270 2024-04-16T11:45:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.114+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a400a370 con 0x7fa0b4103270 2024-04-16T11:45:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.114+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b4115480 con 0x7fa0b4103270 2024-04-16T11:45:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.114+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b4115a40 con 0x7fa0b4103270 2024-04-16T11:45:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.115+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b40760a0 con 0x7fa0b4103270 2024-04-16T11:45:14.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.117+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fa0a4011810 con 0x7fa0b4103270 2024-04-16T11:45:14.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.117+0000 7fa0aaa00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa09806ed30 0x7fa0980711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.118+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa0a408ed40 con 0x7fa0b4103270 2024-04-16T11:45:14.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.118+0000 7fa0abe00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa09806ed30 0x7fa0980711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.119+0000 7fa0abe00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa09806ed30 0x7fa0980711e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00b970 tx=0x7fa09c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:14.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.124+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa0a405b5e0 con 0x7fa0b4103270 2024-04-16T11:45:14.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.378+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b4115e70 con 0x7fa09806ed30 2024-04-16T11:45:14.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.382+0000 7fa0aaa00700 1 -- 172.21.15.44:0/4328456 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fa0b4115e70 con 0x7fa09806ed30 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa09806ed30 msgr2=0x7fa0980711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa09806ed30 0x7fa0980711e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00b970 tx=0x7fa09c009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 msgr2=0x7fa0b4113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 0x7fa0b4113f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c810 tx=0x7fa0a4004470 comp rx=0 tx=0).stop 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 shutdown_connections 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa09806ed30 0x7fa0980711e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0b4103270 0x7fa0b4113f20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 --2- 172.21.15.44:0/4328456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0b4104520 0x7fa0b4114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 >> 172.21.15.44:0/4328456 conn(0x7fa0b4072530 msgr2=0x7fa0b4074ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:14.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 shutdown_connections 2024-04-16T11:45:14.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.385+0000 7fa0b3e00700 1 -- 172.21.15.44:0/4328456 wait complete. 2024-04-16T11:45:14.399 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:45:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 -- 172.21.15.44:0/1578418996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 msgr2=0x7fa0f4103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 --2- 172.21.15.44:0/1578418996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4103830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4004940 tx=0x7fa0e4009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 -- 172.21.15.44:0/1578418996 shutdown_connections 2024-04-16T11:45:14.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 --2- 172.21.15.44:0/1578418996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 0x7fa0f4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 --2- 172.21.15.44:0/1578418996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4103830 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 -- 172.21.15.44:0/1578418996 >> 172.21.15.44:0/1578418996 conn(0x7fa0f4072640 msgr2=0x7fa0f4074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:14.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 -- 172.21.15.44:0/1578418996 shutdown_connections 2024-04-16T11:45:14.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.549+0000 7fa0f9000700 1 -- 172.21.15.44:0/1578418996 wait complete. 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.550+0000 7fa0f9000700 1 Processor -- start 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.550+0000 7fa0f9000700 1 -- start start 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f9000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 0x7fa0f4114540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4114bd0 con 0x7fa0f4103420 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f9000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4119570 con 0x7fa0f41046d0 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47772/0 (socket says 172.21.15.44:47772) 2024-04-16T11:45:14.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1e00700 1 -- 172.21.15.44:0/3216305445 learned_addr learned my addr 172.21.15.44:0/3216305445 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1400700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 0x7fa0f4114540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1e00700 1 -- 172.21.15.44:0/3216305445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 msgr2=0x7fa0f4114540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1e00700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 0x7fa0f4114540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1e00700 1 -- 172.21.15.44:0/3216305445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e4005040 con 0x7fa0f4103420 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.551+0000 7fa0f1400700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 0x7fa0f4114540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.552+0000 7fa0f1e00700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4114000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4008d50 tx=0x7fa0e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:14.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.552+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e400e070 con 0x7fa0f4103420 2024-04-16T11:45:14.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.552+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0e400bbf0 con 0x7fa0f4103420 2024-04-16T11:45:14.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.552+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e401c400 con 0x7fa0f4103420 2024-04-16T11:45:14.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.552+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f4119770 con 0x7fa0f4103420 2024-04-16T11:45:14.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.552+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f4119c80 con 0x7fa0f4103420 2024-04-16T11:45:14.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.553+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f404c710 con 0x7fa0f4103420 2024-04-16T11:45:14.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.554+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fa0e400bd60 con 0x7fa0f4103420 2024-04-16T11:45:14.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.556+0000 7fa0ea000700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa0d806ed30 0x7fa0d80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.556+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa0e4090360 con 0x7fa0f4103420 2024-04-16T11:45:14.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.557+0000 7fa0f1400700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa0d806ed30 0x7fa0d80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.557+0000 7fa0f1400700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa0d806ed30 0x7fa0d80711e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0006fd0 tx=0x7fa0e0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:14.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.561+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa0e405cb90 con 0x7fa0f4103420 2024-04-16T11:45:14.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.813+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0f4108a30 con 0x7fa0d806ed30 2024-04-16T11:45:14.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.816+0000 7fa0ea000700 1 -- 172.21.15.44:0/3216305445 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fa0f4108a30 con 0x7fa0d806ed30 2024-04-16T11:45:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa0d806ed30 msgr2=0x7fa0d80711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa0d806ed30 0x7fa0d80711e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0006fd0 tx=0x7fa0e0008040 comp rx=0 tx=0).stop 2024-04-16T11:45:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 msgr2=0x7fa0f4114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4114000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4008d50 tx=0x7fa0e4009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 shutdown_connections 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa0d806ed30 0x7fa0d80711e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0f4103420 0x7fa0f4114000 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 --2- 172.21.15.44:0/3216305445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0f41046d0 0x7fa0f4114540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.819+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 >> 172.21.15.44:0/3216305445 conn(0x7fa0f4072640 msgr2=0x7fa0f4073d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.820+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 shutdown_connections 2024-04-16T11:45:14.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.820+0000 7fa0f9000700 1 -- 172.21.15.44:0/3216305445 wait complete. 2024-04-16T11:45:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.981+0000 7f067ce00700 1 -- 172.21.15.44:0/2386337484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 msgr2=0x7f06780ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.981+0000 7f067ce00700 1 --2- 172.21.15.44:0/2386337484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f06780ff310 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0668004940 tx=0x7f0668009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.981+0000 7f067ce00700 1 -- 172.21.15.44:0/2386337484 shutdown_connections 2024-04-16T11:45:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.981+0000 7f067ce00700 1 --2- 172.21.15.44:0/2386337484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0678100070 0x7f06781004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.981+0000 7f067ce00700 1 --2- 172.21.15.44:0/2386337484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f06780ff310 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.981+0000 7f067ce00700 1 -- 172.21.15.44:0/2386337484 >> 172.21.15.44:0/2386337484 conn(0x7f06780fa790 msgr2=0x7f06780fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.982+0000 7f067ce00700 1 -- 172.21.15.44:0/2386337484 shutdown_connections 2024-04-16T11:45:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.982+0000 7f067ce00700 1 -- 172.21.15.44:0/2386337484 wait complete. 2024-04-16T11:45:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.983+0000 7f067ce00700 1 Processor -- start 2024-04-16T11:45:14.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.983+0000 7f067ce00700 1 -- start start 2024-04-16T11:45:14.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.983+0000 7f067ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f067810fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.983+0000 7f067ce00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0678100070 0x7f067810ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.983+0000 7f067ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678110540 con 0x7f06780fef00 2024-04-16T11:45:14.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.983+0000 7f067ce00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678110d90 con 0x7f0678100070 2024-04-16T11:45:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f067810fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f067810fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47786/0 (socket says 172.21.15.44:47786) 2024-04-16T11:45:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 -- 172.21.15.44:0/613074063 learned_addr learned my addr 172.21.15.44:0/613074063 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f066fe00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0678100070 0x7f067810ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 -- 172.21.15.44:0/613074063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0678100070 msgr2=0x7f067810ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:14.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0678100070 0x7f067810ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:14.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 -- 172.21.15.44:0/613074063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0668005040 con 0x7f06780fef00 2024-04-16T11:45:14.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.984+0000 7f0677400700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f067810fa00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0668013870 tx=0x7f0668009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:14.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.985+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f066800e070 con 0x7f06780fef00 2024-04-16T11:45:14.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.985+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f066801c400 con 0x7f06780fef00 2024-04-16T11:45:14.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.985+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f066800b630 con 0x7f06780fef00 2024-04-16T11:45:14.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.985+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0678111010 con 0x7f06780fef00 2024-04-16T11:45:14.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.985+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06781114f0 con 0x7f06780fef00 2024-04-16T11:45:14.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.986+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06780fe290 con 0x7f06780fef00 2024-04-16T11:45:14.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.988+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f066801c570 con 0x7f06780fef00 2024-04-16T11:45:14.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.989+0000 7f066f400700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f065c06ed30 0x7f065c0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.990+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0668090200 con 0x7f06780fef00 2024-04-16T11:45:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.990+0000 7f066fe00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f065c06ed30 0x7f065c0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:14.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.991+0000 7f066fe00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f065c06ed30 0x7f065c0711e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0660004600 tx=0x7f066000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:14.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:14.994+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f066805c970 con 0x7f06780fef00 2024-04-16T11:45:15.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.232+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0678111800 con 0x7f065c06ed30 2024-04-16T11:45:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.248+0000 7f066f400700 1 -- 172.21.15.44:0/613074063 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f0678111800 con 0x7f065c06ed30 2024-04-16T11:45:15.245 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (2m) 27s ago 12m 14.0M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (13m) 27s ago 13m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (11m) 4s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (2m) 27s ago 12m 73.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (5m) 27s ago 5m 13.0M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (5m) 27s ago 5m 23.4M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (5m) 4s ago 5m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (5m) 4s ago 5m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (117s) 27s ago 14m 478M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (66s) 4s ago 11m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (45s) 27s ago 14m 30.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (10s) 4s ago 11m 20.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (2m) 27s ago 13m 7918k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:45:15.246 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (2m) 4s ago 11m 7532k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (10m) 27s ago 10m 46.9M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (9m) 27s ago 9m 49.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (9m) 27s ago 9m 46.4M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (8m) 4s ago 8m 46.1M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (8m) 4s ago 8m 44.5M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (7m) 4s ago 7m 46.3M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:45:15.247 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (74s) 27s ago 12m 47.3M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:45:15.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f065c06ed30 msgr2=0x7f065c0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f065c06ed30 0x7f065c0711e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0660004600 tx=0x7f066000b040 comp rx=0 tx=0).stop 2024-04-16T11:45:15.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 msgr2=0x7f067810fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f067810fa00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0668013870 tx=0x7f0668009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:15.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 shutdown_connections 2024-04-16T11:45:15.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f065c06ed30 0x7f065c0711e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06780fef00 0x7f067810fa00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 --2- 172.21.15.44:0/613074063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0678100070 0x7f067810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 >> 172.21.15.44:0/613074063 conn(0x7f06780fa790 msgr2=0x7f06780fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:15.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.251+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 shutdown_connections 2024-04-16T11:45:15.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.252+0000 7f067ce00700 1 -- 172.21.15.44:0/613074063 wait complete. 2024-04-16T11:45:15.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 -- 172.21.15.44:0/955890547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d4071f50 msgr2=0x7f15d4070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 --2- 172.21.15.44:0/955890547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d4071f50 0x7f15d4070460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f15c8004940 tx=0x7f15c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:15.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 -- 172.21.15.44:0/955890547 shutdown_connections 2024-04-16T11:45:15.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 --2- 172.21.15.44:0/955890547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15d40709a0 0x7f15d4070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 --2- 172.21.15.44:0/955890547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d4071f50 0x7f15d4070460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 -- 172.21.15.44:0/955890547 >> 172.21.15.44:0/955890547 conn(0x7f15d40f8530 msgr2=0x7f15d40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:15.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 -- 172.21.15.44:0/955890547 shutdown_connections 2024-04-16T11:45:15.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.406+0000 7f15db800700 1 -- 172.21.15.44:0/955890547 wait complete. 2024-04-16T11:45:15.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.407+0000 7f15db800700 1 Processor -- start 2024-04-16T11:45:15.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.407+0000 7f15db800700 1 -- start start 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15db800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 0x7f15d410fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15db800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15d4071f50 0x7f15d4110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15db800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4110870 con 0x7f15d40709a0 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15db800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4110fb0 con 0x7f15d4071f50 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15d8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 0x7f15d410fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15d8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 0x7f15d410fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47802/0 (socket says 172.21.15.44:47802) 2024-04-16T11:45:15.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15d8e00700 1 -- 172.21.15.44:0/2610244088 learned_addr learned my addr 172.21.15.44:0/2610244088 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15d8e00700 1 -- 172.21.15.44:0/2610244088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15d4071f50 msgr2=0x7f15d4110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15d3e00700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15d4071f50 0x7f15d4110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.408+0000 7f15d8e00700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15d4071f50 0x7f15d4110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15d8e00700 1 -- 172.21.15.44:0/2610244088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8005040 con 0x7f15d40709a0 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15d8e00700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 0x7f15d410fd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f15c800b970 tx=0x7f15c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15c800e040 con 0x7f15d40709a0 2024-04-16T11:45:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15c8004290 con 0x7f15d40709a0 2024-04-16T11:45:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d4111230 con 0x7f15d40709a0 2024-04-16T11:45:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15c801c630 con 0x7f15d40709a0 2024-04-16T11:45:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.409+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d41117f0 con 0x7f15d40709a0 2024-04-16T11:45:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.410+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d405f5f0 con 0x7f15d40709a0 2024-04-16T11:45:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.411+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f15c800bbf0 con 0x7f15d40709a0 2024-04-16T11:45:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.412+0000 7f15d0c00700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f15bc06ed20 0x7f15bc0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.413+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f15c808fdf0 con 0x7f15d40709a0 2024-04-16T11:45:15.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.413+0000 7f15d3e00700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f15bc06ed20 0x7f15bc0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:15.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.413+0000 7f15d3e00700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f15bc06ed20 0x7f15bc0711d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f15c00041d0 tx=0x7f15c0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:15.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.418+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f15c805c560 con 0x7f15d40709a0 2024-04-16T11:45:15.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.746+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f15d4107b90 con 0x7f15d40709a0 2024-04-16T11:45:15.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.747+0000 7f15d0c00700 1 -- 172.21.15.44:0/2610244088 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f15c8013e60 con 0x7f15d40709a0 2024-04-16T11:45:15.744 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:45:15.744 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:45:15.744 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:45:15.745 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f15bc06ed20 msgr2=0x7f15bc0711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f15bc06ed20 0x7f15bc0711d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f15c00041d0 tx=0x7f15c0009040 comp rx=0 tx=0).stop 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 msgr2=0x7f15d410fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 0x7f15d410fd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f15c800b970 tx=0x7f15c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 shutdown_connections 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f15bc06ed20 0x7f15bc0711d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d40709a0 0x7f15d410fd30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 --2- 172.21.15.44:0/2610244088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15d4071f50 0x7f15d4110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 >> 172.21.15.44:0/2610244088 conn(0x7f15d40f8530 msgr2=0x7f15d40f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:15.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 shutdown_connections 2024-04-16T11:45:15.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.750+0000 7f15db800700 1 -- 172.21.15.44:0/2610244088 wait complete. 2024-04-16T11:45:15.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:15 smithi045 ceph-mon[81423]: pgmap v20: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:15.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:15 smithi045 ceph-mon[81423]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:15.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:15 smithi045 ceph-mon[81423]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 -- 172.21.15.44:0/3295974411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 msgr2=0x7f7b0c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 --2- 172.21.15.44:0/3295974411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c070d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7af8004940 tx=0x7f7af8009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 -- 172.21.15.44:0/3295974411 shutdown_connections 2024-04-16T11:45:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 --2- 172.21.15.44:0/3295974411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c070d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 --2- 172.21.15.44:0/3295974411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c071ea0 0x7f7b0c0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 -- 172.21.15.44:0/3295974411 >> 172.21.15.44:0/3295974411 conn(0x7f7b0c0f83d0 msgr2=0x7f7b0c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:15.896 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:15 smithi044 ceph-mon[93020]: pgmap v20: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:15.896 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:15 smithi044 ceph-mon[93020]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:15.896 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:15 smithi044 ceph-mon[93020]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:15.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 -- 172.21.15.44:0/3295974411 shutdown_connections 2024-04-16T11:45:15.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.899+0000 7f7b11400700 1 -- 172.21.15.44:0/3295974411 wait complete. 2024-04-16T11:45:15.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.900+0000 7f7b11400700 1 Processor -- start 2024-04-16T11:45:15.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.900+0000 7f7b11400700 1 -- start start 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b11400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c10fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b11400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c071ea0 0x7f7b0c110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b11400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c1106d0 con 0x7f7b0c0708f0 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b11400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c110e70 con 0x7f7b0c071ea0 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b0be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c10fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b0be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c10fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47832/0 (socket says 172.21.15.44:47832) 2024-04-16T11:45:15.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b0be00700 1 -- 172.21.15.44:0/3551350764 learned_addr learned my addr 172.21.15.44:0/3551350764 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:15.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.901+0000 7f7b0b400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c071ea0 0x7f7b0c110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:15.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b0be00700 1 -- 172.21.15.44:0/3551350764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c071ea0 msgr2=0x7f7b0c110040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:15.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b0be00700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c071ea0 0x7f7b0c110040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:15.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b0be00700 1 -- 172.21.15.44:0/3551350764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8005040 con 0x7f7b0c0708f0 2024-04-16T11:45:15.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b0be00700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c10fb00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00bae0 tx=0x7f7afc009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:15.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7afc00c8c0 con 0x7f7b0c0708f0 2024-04-16T11:45:15.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7afc014410 con 0x7f7b0c0708f0 2024-04-16T11:45:15.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7afc007610 con 0x7f7b0c0708f0 2024-04-16T11:45:15.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.902+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c111150 con 0x7f7b0c0708f0 2024-04-16T11:45:15.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.903+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c111710 con 0x7f7b0c0708f0 2024-04-16T11:45:15.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.904+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c05f5f0 con 0x7f7b0c0708f0 2024-04-16T11:45:15.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.905+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f7afc014580 con 0x7f7b0c0708f0 2024-04-16T11:45:15.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.906+0000 7f7b03e00700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7af006ed20 0x7f7af00711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:15.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.906+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7afc08ea00 con 0x7f7b0c0708f0 2024-04-16T11:45:15.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.907+0000 7f7b0b400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7af006ed20 0x7f7af00711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:15.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.907+0000 7f7b0b400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7af006ed20 0x7f7af00711d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7af8004f90 tx=0x7f7af8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:15.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:15.915+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7afc05b2a0 con 0x7f7b0c0708f0 2024-04-16T11:45:16.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.191+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b0c1119f0 con 0x7f7b0c0708f0 2024-04-16T11:45:16.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.192+0000 7f7b03e00700 1 -- 172.21.15.44:0/3551350764 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f7afc05ae30 con 0x7f7b0c0708f0 2024-04-16T11:45:16.189 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:45:16.190 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:45:16.191 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:16.192 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7af006ed20 msgr2=0x7f7af00711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7af006ed20 0x7f7af00711d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7af8004f90 tx=0x7f7af8009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 msgr2=0x7f7b0c10fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c10fb00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00bae0 tx=0x7f7afc009810 comp rx=0 tx=0).stop 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 shutdown_connections 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7af006ed20 0x7f7af00711d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7b0c0708f0 0x7f7b0c10fb00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 --2- 172.21.15.44:0/3551350764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b0c071ea0 0x7f7b0c110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 >> 172.21.15.44:0/3551350764 conn(0x7f7b0c0f83d0 msgr2=0x7f7b0c0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:16.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 shutdown_connections 2024-04-16T11:45:16.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.195+0000 7f7b11400700 1 -- 172.21.15.44:0/3551350764 wait complete. 2024-04-16T11:45:16.194 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 -- 172.21.15.44:0/1094418253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 msgr2=0x7f08e0102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 --2- 172.21.15.44:0/1094418253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e0102160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08cc004940 tx=0x7f08cc009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 -- 172.21.15.44:0/1094418253 shutdown_connections 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 --2- 172.21.15.44:0/1094418253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e0102160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 --2- 172.21.15.44:0/1094418253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 0x7f08e00ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 -- 172.21.15.44:0/1094418253 >> 172.21.15.44:0/1094418253 conn(0x7f08e00f7270 msgr2=0x7f08e00f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.351+0000 7f08df200700 1 -- 172.21.15.44:0/1094418253 shutdown_connections 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.352+0000 7f08df200700 1 -- 172.21.15.44:0/1094418253 wait complete. 2024-04-16T11:45:16.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.353+0000 7f08df200700 1 Processor -- start 2024-04-16T11:45:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.353+0000 7f08df200700 1 -- start start 2024-04-16T11:45:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.353+0000 7f08df200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 0x7f08e010d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.353+0000 7f08df200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e010dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.353+0000 7f08df200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e010e3e0 con 0x7f08e00ffd40 2024-04-16T11:45:16.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.353+0000 7f08df200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e010ec30 con 0x7f08e00662f0 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e010dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dde00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 0x7f08e010d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dde00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 0x7f08e010d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.44:38528/0 (socket says 172.21.15.44:38528) 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e010dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47854/0 (socket says 172.21.15.44:47854) 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dde00700 1 -- 172.21.15.44:0/1179783568 learned_addr learned my addr 172.21.15.44:0/1179783568 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dd400700 1 -- 172.21.15.44:0/1179783568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 msgr2=0x7f08e010d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dd400700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 0x7f08e010d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dd400700 1 -- 172.21.15.44:0/1179783568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc005040 con 0x7f08e00ffd40 2024-04-16T11:45:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.354+0000 7f08dd400700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e010dde0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00b970 tx=0x7f08cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.355+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08cc00e040 con 0x7f08e00ffd40 2024-04-16T11:45:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.355+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08cc004290 con 0x7f08e00ffd40 2024-04-16T11:45:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.355+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e010eeb0 con 0x7f08e00ffd40 2024-04-16T11:45:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.355+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08cc01c3f0 con 0x7f08e00ffd40 2024-04-16T11:45:16.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.355+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e010f3f0 con 0x7f08e00ffd40 2024-04-16T11:45:16.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.356+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e005f5f0 con 0x7f08e00ffd40 2024-04-16T11:45:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.357+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f08cc00bbf0 con 0x7f08e00ffd40 2024-04-16T11:45:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.358+0000 7f08d6000700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f08c406ed20 0x7f08c40711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.358+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f08cc08fd90 con 0x7f08e00ffd40 2024-04-16T11:45:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.359+0000 7f08dde00700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f08c406ed20 0x7f08c40711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:16.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.359+0000 7f08dde00700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f08c406ed20 0x7f08c40711d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009fd0 tx=0x7f08d000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:16.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.363+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f08cc05c4b0 con 0x7f08e00ffd40 2024-04-16T11:45:16.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.611+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f08e010f7e0 con 0x7f08c406ed20 2024-04-16T11:45:16.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.612+0000 7f08d6000700 1 -- 172.21.15.44:0/1179783568 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f08e010f7e0 con 0x7f08c406ed20 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "4/21 daemons upgraded", 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:45:16.610 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f08c406ed20 msgr2=0x7f08c40711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f08c406ed20 0x7f08c40711d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009fd0 tx=0x7f08d000b040 comp rx=0 tx=0).stop 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 msgr2=0x7f08e010dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e010dde0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00b970 tx=0x7f08cc009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 shutdown_connections 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f08c406ed20 0x7f08c40711d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08e00662f0 0x7f08e010d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 --2- 172.21.15.44:0/1179783568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08e00ffd40 0x7f08e010dde0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 >> 172.21.15.44:0/1179783568 conn(0x7f08e00f7270 msgr2=0x7f08e00f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 shutdown_connections 2024-04-16T11:45:16.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.616+0000 7f08df200700 1 -- 172.21.15.44:0/1179783568 wait complete. 2024-04-16T11:45:16.644 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 236, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82313 2024-04-16T11:45:16.644 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 236, success, outsize: 120 2024-04-16T11:45:16.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 238, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82391 2024-04-16T11:45:16.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 238, success, outsize: 120 2024-04-16T11:45:16.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 240, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82472 2024-04-16T11:45:16.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 240, success, outsize: 120 2024-04-16T11:45:16.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 242, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82559 2024-04-16T11:45:16.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 242, success, outsize: 120 2024-04-16T11:45:16.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 244, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82643 2024-04-16T11:45:16.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 244, success, outsize: 120 2024-04-16T11:45:16.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 -- 172.21.15.44:0/603277213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 msgr2=0x7f8f7c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 --2- 172.21.15.44:0/603277213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c103830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c004940 tx=0x7f8f6c009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:16.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 -- 172.21.15.44:0/603277213 shutdown_connections 2024-04-16T11:45:16.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 --2- 172.21.15.44:0/603277213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f7c1046d0 0x7f8f7c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 --2- 172.21.15.44:0/603277213 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c103830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 -- 172.21.15.44:0/603277213 >> 172.21.15.44:0/603277213 conn(0x7f8f7c072640 msgr2=0x7f8f7c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 -- 172.21.15.44:0/603277213 shutdown_connections 2024-04-16T11:45:16.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.766+0000 7f8f81800700 1 -- 172.21.15.44:0/603277213 wait complete. 2024-04-16T11:45:16.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 246, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82721 2024-04-16T11:45:16.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 246, success, outsize: 120 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.767+0000 7f8f81800700 1 Processor -- start 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.767+0000 7f8f81800700 1 -- start start 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f81800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f81800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f7c1046d0 0x7f8f7c114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f81800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c114a20 con 0x7f8f7c103420 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f81800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c114b90 con 0x7f8f7c1046d0 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47884/0 (socket says 172.21.15.44:47884) 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 -- 172.21.15.44:0/1315439899 learned_addr learned my addr 172.21.15.44:0/1315439899 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7a000700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f7c1046d0 0x7f8f7c114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 -- 172.21.15.44:0/1315439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f7c1046d0 msgr2=0x7f8f7c114420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f7c1046d0 0x7f8f7c114420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:16.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 -- 172.21.15.44:0/1315439899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c005040 con 0x7f8f7c103420 2024-04-16T11:45:16.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.768+0000 7f8f7aa00700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c113ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c00b0d0 tx=0x7f8f6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:16.767 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:16 smithi044 ceph-mon[93020]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:16.767 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:16 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2610244088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:16.767 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:16 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3551350764' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.769+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f6c00e070 con 0x7f8f7c103420 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.769+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f6c00b410 con 0x7f8f7c103420 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.769+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f6c012600 con 0x7f8f7c103420 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.769+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c119590 con 0x7f8f7c103420 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.769+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c119b50 con 0x7f8f7c103420 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.770+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c04c710 con 0x7f8f7c103420 2024-04-16T11:45:16.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.771+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f8f6c00ba80 con 0x7f8f7c103420 2024-04-16T11:45:16.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.773+0000 7f8f72a00700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8f6006ed30 0x7f8f600711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:16.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.773+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8f6c08fb10 con 0x7f8f7c103420 2024-04-16T11:45:16.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.773+0000 7f8f7a000700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8f6006ed30 0x7f8f600711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:16.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.774+0000 7f8f7a000700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8f6006ed30 0x7f8f600711e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8f68004600 tx=0x7f8f6800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:16.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:16.777+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8f6c05c330 con 0x7f8f7c103420 2024-04-16T11:45:16.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 248, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82802 2024-04-16T11:45:16.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 248, success, outsize: 120 2024-04-16T11:45:16.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 250, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82889 2024-04-16T11:45:16.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 250, success, outsize: 120 2024-04-16T11:45:16.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:16 smithi045 ceph-mon[81423]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:16.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:16 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2610244088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:16.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:16 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3551350764' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:45:16.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 252, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82973 2024-04-16T11:45:16.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 252, success, outsize: 120 2024-04-16T11:45:16.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 254, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83051 2024-04-16T11:45:16.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 254, success, outsize: 120 2024-04-16T11:45:16.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 256, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83132 2024-04-16T11:45:16.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 256, success, outsize: 120 2024-04-16T11:45:16.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 258, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83219 2024-04-16T11:45:16.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 258, success, outsize: 120 2024-04-16T11:45:16.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 260, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83303 2024-04-16T11:45:16.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 260, success, outsize: 120 2024-04-16T11:45:16.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 262, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83381 2024-04-16T11:45:16.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 262, success, outsize: 120 2024-04-16T11:45:16.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 264, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83462 2024-04-16T11:45:16.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 264, success, outsize: 120 2024-04-16T11:45:17.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 266, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83549 2024-04-16T11:45:17.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 266, success, outsize: 120 2024-04-16T11:45:17.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 268, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83633 2024-04-16T11:45:17.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 268, success, outsize: 120 2024-04-16T11:45:17.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 270, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83711 2024-04-16T11:45:17.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 270, success, outsize: 120 2024-04-16T11:45:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.087+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8f7c10bf10 con 0x7f8f7c103420 2024-04-16T11:45:17.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.088+0000 7f8f72a00700 1 -- 172.21.15.44:0/1315439899 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f8f6c05bec0 con 0x7f8f7c103420 2024-04-16T11:45:17.085 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8f6006ed30 msgr2=0x7f8f600711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8f6006ed30 0x7f8f600711e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8f68004600 tx=0x7f8f6800b040 comp rx=0 tx=0).stop 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 msgr2=0x7f8f7c113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c113ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c00b0d0 tx=0x7f8f6c009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 shutdown_connections 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8f6006ed30 0x7f8f600711e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8f7c103420 0x7f8f7c113ee0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 --2- 172.21.15.44:0/1315439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f7c1046d0 0x7f8f7c114420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:17.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 >> 172.21.15.44:0/1315439899 conn(0x7f8f7c072640 msgr2=0x7f8f7c073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 shutdown_connections 2024-04-16T11:45:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:17.092+0000 7f8f81800700 1 -- 172.21.15.44:0/1315439899 wait complete. 2024-04-16T11:45:17.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 272, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83792 2024-04-16T11:45:17.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 272, success, outsize: 120 2024-04-16T11:45:17.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 274, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83879 2024-04-16T11:45:17.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 274, success, outsize: 120 2024-04-16T11:45:17.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 276, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83963 2024-04-16T11:45:17.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 276, success, outsize: 120 2024-04-16T11:45:17.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 278, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84041 2024-04-16T11:45:17.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 278, success, outsize: 120 2024-04-16T11:45:17.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 280, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84122 2024-04-16T11:45:17.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 280, success, outsize: 120 2024-04-16T11:45:17.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 282, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84209 2024-04-16T11:45:17.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 282, success, outsize: 120 2024-04-16T11:45:17.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 284, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84293 2024-04-16T11:45:17.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 284, success, outsize: 120 2024-04-16T11:45:17.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 286, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84371 2024-04-16T11:45:17.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 286, success, outsize: 120 2024-04-16T11:45:17.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 288, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84452 2024-04-16T11:45:17.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 288, success, outsize: 120 2024-04-16T11:45:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 290, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84539 2024-04-16T11:45:17.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 290, success, outsize: 120 2024-04-16T11:45:17.814 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:17 smithi045 ceph-mon[81423]: pgmap v21: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:17.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:17 smithi045 ceph-mon[81423]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:17.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:17 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1315439899' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:45:17.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:17 smithi044 ceph-mon[93020]: pgmap v21: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:17.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:17 smithi044 ceph-mon[93020]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:17.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:17 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1315439899' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:45:19.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: pgmap v22: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:19.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:19.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:19.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:19.667 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:45:19.668 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:19.668 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: Reconfiguring mon.smithi044 (monmap changed)... 2024-04-16T11:45:19.668 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:45:19.668 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:45:19.668 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:19.668 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:19 smithi044 ceph-mon[93020]: Reconfiguring daemon mon.smithi044 on smithi044 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: pgmap v22: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: Reconfiguring mon.smithi044 (monmap changed)... 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:19 smithi045 ceph-mon[81423]: Reconfiguring daemon mon.smithi044 on smithi044 2024-04-16T11:45:21.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:21 smithi044 ceph-mon[93020]: pgmap v23: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:21.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:21.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:21.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:45:21.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:45:21.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:21.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:21 smithi045 ceph-mon[81423]: pgmap v23: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi044.sczobv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:45:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:45:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:22 smithi044 ceph-mon[93020]: Reconfiguring mgr.smithi044.sczobv (monmap changed)... 2024-04-16T11:45:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:22 smithi044 ceph-mon[93020]: Reconfiguring daemon mgr.smithi044.sczobv on smithi044 2024-04-16T11:45:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:22 smithi045 ceph-mon[81423]: Reconfiguring mgr.smithi044.sczobv (monmap changed)... 2024-04-16T11:45:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:22 smithi045 ceph-mon[81423]: Reconfiguring daemon mgr.smithi044.sczobv on smithi044 2024-04-16T11:45:23.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:23 smithi044 ceph-mon[93020]: pgmap v24: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:23.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:23 smithi045 ceph-mon[81423]: pgmap v24: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:24.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:24 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:24.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:24 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:24.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:24 smithi044 ceph-mon[93020]: Reconfiguring crash.smithi044 (monmap changed)... 2024-04-16T11:45:24.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:24 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:45:24.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:24 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:24.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:24 smithi044 ceph-mon[93020]: Reconfiguring daemon crash.smithi044 on smithi044 2024-04-16T11:45:24.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:24 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:24 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:24 smithi045 ceph-mon[81423]: Reconfiguring crash.smithi044 (monmap changed)... 2024-04-16T11:45:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:24 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:45:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:24 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:24.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:24 smithi045 ceph-mon[81423]: Reconfiguring daemon crash.smithi044 on smithi044 2024-04-16T11:45:25.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:25 smithi045 ceph-mon[81423]: pgmap v25: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:25.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:25.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:25 smithi044 ceph-mon[93020]: pgmap v25: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:25.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:26.939 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:26.939 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:26.939 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:26 smithi044 ceph-mon[93020]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T11:45:26.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T11:45:26.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:26.940 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:26 smithi044 ceph-mon[93020]: Reconfiguring daemon osd.0 on smithi044 2024-04-16T11:45:27.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:27.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:27.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:26 smithi045 ceph-mon[81423]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T11:45:27.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T11:45:27.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:27.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:26 smithi045 ceph-mon[81423]: Reconfiguring daemon osd.0 on smithi044 2024-04-16T11:45:28.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:27 smithi045 ceph-mon[81423]: pgmap v26: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:28.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:27 smithi044 ceph-mon[93020]: pgmap v26: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: pgmap v27: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:29.528 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:29 smithi044 ceph-mon[93020]: Reconfiguring daemon osd.1 on smithi044 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: pgmap v27: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:29 smithi045 ceph-mon[81423]: Reconfiguring daemon osd.1 on smithi044 2024-04-16T11:45:32.214 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:31 smithi044 ceph-mon[93020]: pgmap v28: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:32.214 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:32.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:32.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T11:45:32.215 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:32.285 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:31 smithi045 ceph-mon[81423]: pgmap v28: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:32.285 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:32.285 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:32.285 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T11:45:32.285 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:33.181 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:32 smithi044 ceph-mon[93020]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T11:45:33.182 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:32 smithi044 ceph-mon[93020]: Reconfiguring daemon osd.2 on smithi044 2024-04-16T11:45:33.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:32 smithi045 ceph-mon[81423]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T11:45:33.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:32 smithi045 ceph-mon[81423]: Reconfiguring daemon osd.2 on smithi044 2024-04-16T11:45:33.993 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:33 smithi045 ceph-mon[81423]: pgmap v29: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:33.993 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:33.993 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:33.993 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:33.993 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:34.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:33 smithi044 ceph-mon[93020]: pgmap v29: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:34.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:34.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:34.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:35.176 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:34 smithi044 ceph-mon[93020]: Reconfiguring mds.cephfs.smithi044.ufhhlc (monmap changed)... 2024-04-16T11:45:35.176 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:34 smithi044 ceph-mon[93020]: Reconfiguring daemon mds.cephfs.smithi044.ufhhlc on smithi044 2024-04-16T11:45:35.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:34 smithi045 ceph-mon[81423]: Reconfiguring mds.cephfs.smithi044.ufhhlc (monmap changed)... 2024-04-16T11:45:35.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:34 smithi045 ceph-mon[81423]: Reconfiguring daemon mds.cephfs.smithi044.ufhhlc on smithi044 2024-04-16T11:45:36.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:35 smithi044 ceph-mon[93020]: pgmap v30: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:36.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:36.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:36.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:36.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:35 smithi045 ceph-mon[81423]: pgmap v30: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:36.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:36.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:36.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:37.100 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:36 smithi044 ceph-mon[93020]: Reconfiguring mds.cephfs.smithi044.hlafhv (monmap changed)... 2024-04-16T11:45:37.100 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:37.100 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:36 smithi044 ceph-mon[93020]: Reconfiguring daemon mds.cephfs.smithi044.hlafhv on smithi044 2024-04-16T11:45:37.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:36 smithi045 ceph-mon[81423]: Reconfiguring mds.cephfs.smithi044.hlafhv (monmap changed)... 2024-04-16T11:45:37.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:37.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:36 smithi045 ceph-mon[81423]: Reconfiguring daemon mds.cephfs.smithi044.hlafhv on smithi044 2024-04-16T11:45:38.120 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:37 smithi044 ceph-mon[93020]: pgmap v31: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:38.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:37 smithi045 ceph-mon[81423]: pgmap v31: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: pgmap v32: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: Reconfiguring crash.smithi045 (monmap changed)... 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:39.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:39 smithi045 ceph-mon[81423]: Reconfiguring daemon crash.smithi045 on smithi045 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: pgmap v32: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: Reconfiguring crash.smithi045 (monmap changed)... 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:39.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:39 smithi044 ceph-mon[93020]: Reconfiguring daemon crash.smithi045 on smithi045 2024-04-16T11:45:40.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:40.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: pgmap v33: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: Reconfiguring mgr.smithi045.sdlwif (monmap changed)... 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:45:41.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:41.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:41 smithi045 ceph-mon[81423]: Reconfiguring daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: pgmap v33: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: Reconfiguring mgr.smithi045.sdlwif (monmap changed)... 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.sdlwif", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:41.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:41 smithi044 ceph-mon[93020]: Reconfiguring daemon mgr.smithi045.sdlwif on smithi045 2024-04-16T11:45:43.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:43 smithi044 ceph-mon[93020]: pgmap v34: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:43.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:43 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:43.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:43 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:43.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:43 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:45:43.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:43 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:45:43.779 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:43 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:43.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:43 smithi045 ceph-mon[81423]: pgmap v34: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:43.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:43 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:43.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:43 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:43.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:43 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T11:45:43.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:43 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T11:45:43.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:43 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:44.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:44 smithi045 ceph-mon[81423]: Reconfiguring mon.smithi045 (monmap changed)... 2024-04-16T11:45:44.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:44 smithi045 ceph-mon[81423]: Reconfiguring daemon mon.smithi045 on smithi045 2024-04-16T11:45:44.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:44 smithi044 ceph-mon[93020]: Reconfiguring mon.smithi045 (monmap changed)... 2024-04-16T11:45:44.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:44 smithi044 ceph-mon[93020]: Reconfiguring daemon mon.smithi045 on smithi045 2024-04-16T11:45:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:45 smithi045 ceph-mon[81423]: pgmap v35: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T11:45:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:45.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:45 smithi044 ceph-mon[93020]: pgmap v35: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:45.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:45.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:45.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T11:45:45.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:46.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:46 smithi044 ceph-mon[93020]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T11:45:46.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:46 smithi044 ceph-mon[93020]: Reconfiguring daemon osd.3 on smithi045 2024-04-16T11:45:46.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:46 smithi045 ceph-mon[81423]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T11:45:46.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:46 smithi045 ceph-mon[81423]: Reconfiguring daemon osd.3 on smithi045 2024-04-16T11:45:47.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 -- 172.21.15.44:0/372934738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 msgr2=0x7f5368104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 --2- 172.21.15.44:0/372934738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368104b40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5358004940 tx=0x7f5358009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 -- 172.21.15.44:0/372934738 shutdown_connections 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 --2- 172.21.15.44:0/372934738 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368104b40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 --2- 172.21.15.44:0/372934738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5368103420 0x7f5368103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 -- 172.21.15.44:0/372934738 >> 172.21.15.44:0/372934738 conn(0x7f5368072640 msgr2=0x7f5368074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.264+0000 7f5370800700 1 -- 172.21.15.44:0/372934738 shutdown_connections 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.265+0000 7f5370800700 1 -- 172.21.15.44:0/372934738 wait complete. 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.266+0000 7f5370800700 1 Processor -- start 2024-04-16T11:45:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.266+0000 7f5370800700 1 -- start start 2024-04-16T11:45:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.266+0000 7f5370800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5368103420 0x7f5368113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.266+0000 7f5370800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.266+0000 7f5370800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368114a10 con 0x7f53681046d0 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.266+0000 7f5370800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368114b80 con 0x7f5368103420 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368114410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34948/0 (socket says 172.21.15.44:34948) 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536de00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5368103420 0x7f5368113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 -- 172.21.15.44:0/2905544646 learned_addr learned my addr 172.21.15.44:0/2905544646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 -- 172.21.15.44:0/2905544646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5368103420 msgr2=0x7f5368113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5368103420 0x7f5368113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 -- 172.21.15.44:0/2905544646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358005040 con 0x7f53681046d0 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.267+0000 7f536d400700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368114410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5358013870 tx=0x7f5358009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.268+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f535800e070 con 0x7f53681046d0 2024-04-16T11:45:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.268+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5358013de0 con 0x7f53681046d0 2024-04-16T11:45:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.268+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5368119580 con 0x7f53681046d0 2024-04-16T11:45:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.268+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f535801cea0 con 0x7f53681046d0 2024-04-16T11:45:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.268+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5368119a60 con 0x7f53681046d0 2024-04-16T11:45:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.269+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536804c710 con 0x7f53681046d0 2024-04-16T11:45:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.270+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f535801c690 con 0x7f53681046d0 2024-04-16T11:45:47.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.271+0000 7f5362000700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f535006ed30 0x7f53500711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:47.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.271+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f53580901a0 con 0x7f53681046d0 2024-04-16T11:45:47.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.272+0000 7f536de00700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f535006ed30 0x7f53500711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:47.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.272+0000 7f536de00700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f535006ed30 0x7f53500711e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f535c009b80 tx=0x7f535c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:47.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.278+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f535805c910 con 0x7f53681046d0 2024-04-16T11:45:47.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.532+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53681089a0 con 0x7f535006ed30 2024-04-16T11:45:47.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.534+0000 7f5362000700 1 -- 172.21.15.44:0/2905544646 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f53681089a0 con 0x7f535006ed30 2024-04-16T11:45:47.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f535006ed30 msgr2=0x7f53500711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f535006ed30 0x7f53500711e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f535c009b80 tx=0x7f535c00b040 comp rx=0 tx=0).stop 2024-04-16T11:45:47.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 msgr2=0x7f5368114410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368114410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5358013870 tx=0x7f5358009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:47.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 shutdown_connections 2024-04-16T11:45:47.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f535006ed30 0x7f53500711e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5368103420 0x7f5368113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 --2- 172.21.15.44:0/2905544646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53681046d0 0x7f5368114410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 >> 172.21.15.44:0/2905544646 conn(0x7f5368072640 msgr2=0x7f5368073c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:47.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 shutdown_connections 2024-04-16T11:45:47.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.537+0000 7f5370800700 1 -- 172.21.15.44:0/2905544646 wait complete. 2024-04-16T11:45:47.552 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 -- 172.21.15.44:0/1792081998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c103420 msgr2=0x7f555c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 --2- 172.21.15.44:0/1792081998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c103420 0x7f555c103830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f554c009ce0 tx=0x7f554c004d10 comp rx=0 tx=0).stop 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 -- 172.21.15.44:0/1792081998 shutdown_connections 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 --2- 172.21.15.44:0/1792081998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f555c1046d0 0x7f555c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 --2- 172.21.15.44:0/1792081998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c103420 0x7f555c103830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 -- 172.21.15.44:0/1792081998 >> 172.21.15.44:0/1792081998 conn(0x7f555c072640 msgr2=0x7f555c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:47.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 -- 172.21.15.44:0/1792081998 shutdown_connections 2024-04-16T11:45:47.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.695+0000 7f5562e00700 1 -- 172.21.15.44:0/1792081998 wait complete. 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.696+0000 7f5562e00700 1 Processor -- start 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.696+0000 7f5562e00700 1 -- start start 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5562e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f555c103420 0x7f555c114110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5562e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 0x7f555c114650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5562e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c114c50 con 0x7f555c1046d0 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5562e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c114dc0 con 0x7f555c103420 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5553400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 0x7f555c114650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5553400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 0x7f555c114650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34958/0 (socket says 172.21.15.44:34958) 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5553400700 1 -- 172.21.15.44:0/401687575 learned_addr learned my addr 172.21.15.44:0/401687575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:47.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f555be00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f555c103420 0x7f555c114110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5553400700 1 -- 172.21.15.44:0/401687575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f555c103420 msgr2=0x7f555c114110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5553400700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f555c103420 0x7f555c114110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.697+0000 7f5553400700 1 -- 172.21.15.44:0/401687575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c009940 con 0x7f555c1046d0 2024-04-16T11:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.698+0000 7f5553400700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 0x7f555c114650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5544002a50 tx=0x7f55440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.698+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5544007cc0 con 0x7f555c1046d0 2024-04-16T11:45:47.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.698+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f554400f040 con 0x7f555c1046d0 2024-04-16T11:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.698+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c119820 con 0x7f555c1046d0 2024-04-16T11:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.698+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554400da30 con 0x7f555c1046d0 2024-04-16T11:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.698+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c119de0 con 0x7f555c1046d0 2024-04-16T11:45:47.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.699+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c04c710 con 0x7f555c1046d0 2024-04-16T11:45:47.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.700+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f554400db90 con 0x7f555c1046d0 2024-04-16T11:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.701+0000 7f5558c00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f554006ed30 0x7f55400711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.701+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f554408e8f0 con 0x7f555c1046d0 2024-04-16T11:45:47.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.701+0000 7f555be00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f554006ed30 0x7f55400711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:47.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.702+0000 7f555be00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f554006ed30 0x7f55400711e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f554c015040 tx=0x7f554c004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:47.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.706+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f554405af60 con 0x7f555c1046d0 2024-04-16T11:45:47.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.958+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f555c108ae0 con 0x7f554006ed30 2024-04-16T11:45:47.957 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:47 smithi044 ceph-mon[93020]: pgmap v36: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:47.957 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:47.957 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:47.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.961+0000 7f5558c00700 1 -- 172.21.15.44:0/401687575 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f555c108ae0 con 0x7f554006ed30 2024-04-16T11:45:47.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f554006ed30 msgr2=0x7f55400711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f554006ed30 0x7f55400711e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f554c015040 tx=0x7f554c004d10 comp rx=0 tx=0).stop 2024-04-16T11:45:47.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 msgr2=0x7f555c114650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:47.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 0x7f555c114650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5544002a50 tx=0x7f55440046b0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 shutdown_connections 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f554006ed30 0x7f55400711e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f555c103420 0x7f555c114110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 --2- 172.21.15.44:0/401687575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f555c1046d0 0x7f555c114650 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.965+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 >> 172.21.15.44:0/401687575 conn(0x7f555c072640 msgr2=0x7f555c073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.966+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 shutdown_connections 2024-04-16T11:45:47.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:47.966+0000 7f5562e00700 1 -- 172.21.15.44:0/401687575 wait complete. 2024-04-16T11:45:48.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:47 smithi045 ceph-mon[81423]: pgmap v36: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:48.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 -- 172.21.15.44:0/3913199200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f41002b0 msgr2=0x7fb3f4100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/3913199200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f41002b0 0x7fb3f4100720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc004940 tx=0x7fb3dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:48.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 -- 172.21.15.44:0/3913199200 shutdown_connections 2024-04-16T11:45:48.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/3913199200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f41002b0 0x7fb3f4100720 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/3913199200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3f40ff000 0x7fb3f40ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 -- 172.21.15.44:0/3913199200 >> 172.21.15.44:0/3913199200 conn(0x7fb3f40fa950 msgr2=0x7fb3f40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:48.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 -- 172.21.15.44:0/3913199200 shutdown_connections 2024-04-16T11:45:48.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.127+0000 7fb3f9c00700 1 -- 172.21.15.44:0/3913199200 wait complete. 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.128+0000 7fb3f9c00700 1 Processor -- start 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.128+0000 7fb3f9c00700 1 -- start start 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 0x7fb3f410fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f9c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3f41002b0 0x7fb3f4110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f9c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4110770 con 0x7fb3f40ff000 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f9c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4110fc0 con 0x7fb3f41002b0 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 0x7fb3f410fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 0x7fb3f410fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34974/0 (socket says 172.21.15.44:34974) 2024-04-16T11:45:48.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f2a00700 1 -- 172.21.15.44:0/66372794 learned_addr learned my addr 172.21.15.44:0/66372794 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:48.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3ea000700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3f41002b0 0x7fb3f4110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:48.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f2a00700 1 -- 172.21.15.44:0/66372794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3f41002b0 msgr2=0x7fb3f4110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f2a00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3f41002b0 0x7fb3f4110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.129+0000 7fb3f2a00700 1 -- 172.21.15.44:0/66372794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc005040 con 0x7fb3f40ff000 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.130+0000 7fb3f2a00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 0x7fb3f410fc30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400cac0 tx=0x7fb3e4007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.130+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3e400aa50 con 0x7fb3f40ff000 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.130+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3e400abb0 con 0x7fb3f40ff000 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.130+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f41112a0 con 0x7fb3f40ff000 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.130+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3e40115e0 con 0x7fb3f40ff000 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.130+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f41117e0 con 0x7fb3f40ff000 2024-04-16T11:45:48.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.131+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f405f5f0 con 0x7fb3f40ff000 2024-04-16T11:45:48.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.132+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb3e400ad20 con 0x7fb3f40ff000 2024-04-16T11:45:48.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.133+0000 7fb3eb400700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb3d806ed20 0x7fb3d80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:48.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.133+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb3e408f050 con 0x7fb3f40ff000 2024-04-16T11:45:48.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.134+0000 7fb3ea000700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb3d806ed20 0x7fb3d80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:48.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.134+0000 7fb3ea000700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb3d806ed20 0x7fb3d80711d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00b970 tx=0x7fb3dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:48.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.139+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb3e405b8f0 con 0x7fb3f40ff000 2024-04-16T11:45:48.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.384+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3f4111b00 con 0x7fb3d806ed20 2024-04-16T11:45:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.396+0000 7fb3eb400700 1 -- 172.21.15.44:0/66372794 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fb3f4111b00 con 0x7fb3d806ed20 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (2m) 60s ago 13m 14.0M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (13m) 60s ago 13m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 284ce1cef0a0 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (12m) 38s ago 12m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 2eee6258e743 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (2m) 60s ago 12m 73.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (6m) 60s ago 6m 13.0M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (6m) 60s ago 6m 23.4M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (6m) 38s ago 6m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:45:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (6m) 38s ago 6m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (2m) 60s ago 15m 478M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (99s) 38s ago 12m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (78s) 60s ago 15m 30.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (43s) 38s ago 11m 20.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (3m) 60s ago 13m 7918k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (2m) 38s ago 12m 7532k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (11m) 60s ago 11m 46.9M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (10m) 60s ago 10m 49.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (9m) 60s ago 9m 46.4M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (9m) 38s ago 9m 46.1M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (8m) 38s ago 8m 44.5M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (8m) 38s ago 8m 46.3M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:45:48.396 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (107s) 60s ago 12m 47.3M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:45:48.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.399+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb3d806ed20 msgr2=0x7fb3d80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.399+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb3d806ed20 0x7fb3d80711d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00b970 tx=0x7fb3dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:48.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.399+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 msgr2=0x7fb3f410fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.399+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 0x7fb3f410fc30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400cac0 tx=0x7fb3e4007ee0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.399+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 shutdown_connections 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.400+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb3d806ed20 0x7fb3d80711d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.400+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3f40ff000 0x7fb3f410fc30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.400+0000 7fb3f9c00700 1 --2- 172.21.15.44:0/66372794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3f41002b0 0x7fb3f4110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.400+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 >> 172.21.15.44:0/66372794 conn(0x7fb3f40fa950 msgr2=0x7fb3f40fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.400+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 shutdown_connections 2024-04-16T11:45:48.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.400+0000 7fb3f9c00700 1 -- 172.21.15.44:0/66372794 wait complete. 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 -- 172.21.15.44:0/874231834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 msgr2=0x7f39080fe500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 --2- 172.21.15.44:0/874231834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f39080fe500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f38f4004940 tx=0x7f38f4009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 -- 172.21.15.44:0/874231834 shutdown_connections 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 --2- 172.21.15.44:0/874231834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f39080fe500 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 --2- 172.21.15.44:0/874231834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39080fcf20 0x7f39080fd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 -- 172.21.15.44:0/874231834 >> 172.21.15.44:0/874231834 conn(0x7f39080f85c0 msgr2=0x7f39080faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 -- 172.21.15.44:0/874231834 shutdown_connections 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.557+0000 7f3910000700 1 -- 172.21.15.44:0/874231834 wait complete. 2024-04-16T11:45:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.558+0000 7f3910000700 1 Processor -- start 2024-04-16T11:45:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.558+0000 7f3910000700 1 -- start start 2024-04-16T11:45:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f3910000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39080fcf20 0x7f3908071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f3910000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f3908070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f3910000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080708a0 con 0x7f39080fe090 2024-04-16T11:45:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f3910000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080709e0 con 0x7f39080fcf20 2024-04-16T11:45:48.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f3908070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:48.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f3908070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34984/0 (socket says 172.21.15.44:34984) 2024-04-16T11:45:48.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390cc00700 1 -- 172.21.15.44:0/2344008814 learned_addr learned my addr 172.21.15.44:0/2344008814 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:48.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390d600700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39080fcf20 0x7f3908071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:48.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390cc00700 1 -- 172.21.15.44:0/2344008814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39080fcf20 msgr2=0x7f3908071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390cc00700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39080fcf20 0x7f3908071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.559+0000 7f390cc00700 1 -- 172.21.15.44:0/2344008814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4005040 con 0x7f39080fe090 2024-04-16T11:45:48.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.560+0000 7f390cc00700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f3908070310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f38f400b3b0 tx=0x7f38f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:48.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.560+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f400e040 con 0x7f39080fe090 2024-04-16T11:45:48.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.560+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38f4004290 con 0x7f39080fe090 2024-04-16T11:45:48.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.560+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f401c630 con 0x7f39080fe090 2024-04-16T11:45:48.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.560+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3908070c60 con 0x7f39080fe090 2024-04-16T11:45:48.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.560+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3908115b00 con 0x7f39080fe090 2024-04-16T11:45:48.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.562+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390805f5f0 con 0x7f39080fe090 2024-04-16T11:45:48.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.563+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f38f400bbf0 con 0x7f39080fe090 2024-04-16T11:45:48.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.564+0000 7f3905600700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38f006ed20 0x7f38f00711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:48.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.564+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f38f408fe00 con 0x7f39080fe090 2024-04-16T11:45:48.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.565+0000 7f390d600700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38f006ed20 0x7f38f00711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:48.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.565+0000 7f390d600700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38f006ed20 0x7f38f00711d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009fd0 tx=0x7f38fc009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:48.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.569+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f38f4058b00 con 0x7f39080fe090 2024-04-16T11:45:48.670 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:48 smithi044 ceph-mon[93020]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:48.670 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:48 smithi044 ceph-mon[93020]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T11:45:48.670 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T11:45:48.670 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:48.670 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:48 smithi044 ceph-mon[93020]: Reconfiguring daemon osd.4 on smithi045 2024-04-16T11:45:48.797 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:48 smithi045 ceph-mon[81423]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:48.797 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:48 smithi045 ceph-mon[81423]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T11:45:48.797 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:48 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T11:45:48.798 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:48 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:48.798 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:48 smithi045 ceph-mon[81423]: Reconfiguring daemon osd.4 on smithi045 2024-04-16T11:45:48.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.898+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3908105930 con 0x7f39080fe090 2024-04-16T11:45:48.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.898+0000 7f3905600700 1 -- 172.21.15.44:0/2344008814 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f38f405c1c0 con 0x7f39080fe090 2024-04-16T11:45:48.896 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T11:45:48.897 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:45:48.898 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:45:48.898 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:45:48.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38f006ed20 msgr2=0x7f38f00711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38f006ed20 0x7f38f00711d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009fd0 tx=0x7f38fc009f20 comp rx=0 tx=0).stop 2024-04-16T11:45:48.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 msgr2=0x7f3908070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f3908070310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f38f400b3b0 tx=0x7f38f4009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 shutdown_connections 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38f006ed20 0x7f38f00711d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39080fcf20 0x7f3908071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 --2- 172.21.15.44:0/2344008814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39080fe090 0x7f3908070310 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 >> 172.21.15.44:0/2344008814 conn(0x7f39080f85c0 msgr2=0x7f39080f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 shutdown_connections 2024-04-16T11:45:48.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:48.901+0000 7f3910000700 1 -- 172.21.15.44:0/2344008814 wait complete. 2024-04-16T11:45:49.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 -- 172.21.15.44:0/205134017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 msgr2=0x7fd6640fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 --2- 172.21.15.44:0/205134017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd6640fd030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd654004940 tx=0x7fd654009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:49.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 -- 172.21.15.44:0/205134017 shutdown_connections 2024-04-16T11:45:49.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 --2- 172.21.15.44:0/205134017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6640fded0 0x7fd6640fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 --2- 172.21.15.44:0/205134017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd6640fd030 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 -- 172.21.15.44:0/205134017 >> 172.21.15.44:0/205134017 conn(0x7fd6640f83a0 msgr2=0x7fd6640fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:49.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 -- 172.21.15.44:0/205134017 shutdown_connections 2024-04-16T11:45:49.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.062+0000 7fd668e00700 1 -- 172.21.15.44:0/205134017 wait complete. 2024-04-16T11:45:49.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.063+0000 7fd668e00700 1 Processor -- start 2024-04-16T11:45:49.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.063+0000 7fd668e00700 1 -- start start 2024-04-16T11:45:49.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd668e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd66410d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd668e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6640fded0 0x7fd66410dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd668e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66410e2f0 con 0x7fd6640fcc20 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd668e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66410eb40 con 0x7fd6640fded0 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd663400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd66410d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd663400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd66410d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35000/0 (socket says 172.21.15.44:35000) 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd663400700 1 -- 172.21.15.44:0/971507004 learned_addr learned my addr 172.21.15.44:0/971507004 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd662a00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6640fded0 0x7fd66410dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd663400700 1 -- 172.21.15.44:0/971507004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6640fded0 msgr2=0x7fd66410dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd663400700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6640fded0 0x7fd66410dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.064+0000 7fd663400700 1 -- 172.21.15.44:0/971507004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd654005040 con 0x7fd6640fcc20 2024-04-16T11:45:49.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.065+0000 7fd663400700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd66410d7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd654013870 tx=0x7fd654009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:49.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.065+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd654021070 con 0x7fd6640fcc20 2024-04-16T11:45:49.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.065+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd65401c400 con 0x7fd6640fcc20 2024-04-16T11:45:49.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.065+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd65400e070 con 0x7fd6640fcc20 2024-04-16T11:45:49.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.065+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66410edc0 con 0x7fd6640fcc20 2024-04-16T11:45:49.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.065+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66410f300 con 0x7fd6640fcc20 2024-04-16T11:45:49.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.066+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6640fc020 con 0x7fd6640fcc20 2024-04-16T11:45:49.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.067+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fd65401ca20 con 0x7fd6640fcc20 2024-04-16T11:45:49.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.068+0000 7fd65b400700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd64806ed30 0x7fd6480711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.069+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd65408fbd0 con 0x7fd6640fcc20 2024-04-16T11:45:49.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.069+0000 7fd662a00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd64806ed30 0x7fd6480711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.070+0000 7fd662a00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd64806ed30 0x7fd6480711e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd650004600 tx=0x7fd65000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:49.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.073+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd65405c400 con 0x7fd6640fcc20 2024-04-16T11:45:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.359+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd664105710 con 0x7fd6640fcc20 2024-04-16T11:45:49.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.360+0000 7fd65b400700 1 -- 172.21.15.44:0/971507004 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7fd654004490 con 0x7fd6640fcc20 2024-04-16T11:45:49.358 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:45:49.358 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:45:49.358 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:45:49.358 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:45:49.358 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:49.358 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:45:49.359 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:49.360 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:49.361 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:45:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd64806ed30 msgr2=0x7fd6480711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd64806ed30 0x7fd6480711e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd650004600 tx=0x7fd65000b040 comp rx=0 tx=0).stop 2024-04-16T11:45:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 msgr2=0x7fd66410d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd66410d7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd654013870 tx=0x7fd654009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 shutdown_connections 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd64806ed30 0x7fd6480711e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6640fcc20 0x7fd66410d7b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 --2- 172.21.15.44:0/971507004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6640fded0 0x7fd66410dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.364+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 >> 172.21.15.44:0/971507004 conn(0x7fd6640f83a0 msgr2=0x7fd6640f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.365+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 shutdown_connections 2024-04-16T11:45:49.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.365+0000 7fd668e00700 1 -- 172.21.15.44:0/971507004 wait complete. 2024-04-16T11:45:49.364 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.527+0000 7efd09200700 1 -- 172.21.15.44:0/106170572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04104520 msgr2=0x7efd04104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.527+0000 7efd09200700 1 --2- 172.21.15.44:0/106170572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04104520 0x7efd04104990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efcf0004940 tx=0x7efcf0009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.527+0000 7efd09200700 1 -- 172.21.15.44:0/106170572 shutdown_connections 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.527+0000 7efd09200700 1 --2- 172.21.15.44:0/106170572 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04104520 0x7efd04104990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.527+0000 7efd09200700 1 --2- 172.21.15.44:0/106170572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04103270 0x7efd04103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.527+0000 7efd09200700 1 -- 172.21.15.44:0/106170572 >> 172.21.15.44:0/106170572 conn(0x7efd04072530 msgr2=0x7efd04074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.528+0000 7efd09200700 1 -- 172.21.15.44:0/106170572 shutdown_connections 2024-04-16T11:45:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.528+0000 7efd09200700 1 -- 172.21.15.44:0/106170572 wait complete. 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.528+0000 7efd09200700 1 Processor -- start 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd09200700 1 -- start start 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd09200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 0x7efd04113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd09200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04104520 0x7efd04114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd09200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04114a60 con 0x7efd04103270 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd09200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd041151a0 con 0x7efd04104520 2024-04-16T11:45:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd03400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 0x7efd04113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd03400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 0x7efd04113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35018/0 (socket says 172.21.15.44:35018) 2024-04-16T11:45:49.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.529+0000 7efd03400700 1 -- 172.21.15.44:0/3094141227 learned_addr learned my addr 172.21.15.44:0/3094141227 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efd02a00700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04104520 0x7efd04114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efd03400700 1 -- 172.21.15.44:0/3094141227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04104520 msgr2=0x7efd04114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efd03400700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04104520 0x7efd04114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efd03400700 1 -- 172.21.15.44:0/3094141227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf0005040 con 0x7efd04103270 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efd02a00700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04104520 0x7efd04114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efd03400700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 0x7efd04113f20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efcf4002a50 tx=0x7efcf40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcf4007cc0 con 0x7efd04103270 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.530+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcf400f040 con 0x7efd04103270 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.531+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd04115480 con 0x7efd04103270 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.531+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcf4009640 con 0x7efd04103270 2024-04-16T11:45:49.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.531+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd04115a40 con 0x7efd04103270 2024-04-16T11:45:49.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.532+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd040760a0 con 0x7efd04103270 2024-04-16T11:45:49.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.533+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7efcf40097a0 con 0x7efd04103270 2024-04-16T11:45:49.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.534+0000 7efcfb400700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efce806ed30 0x7efce80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.534+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efcf408e880 con 0x7efd04103270 2024-04-16T11:45:49.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.534+0000 7efd02a00700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efce806ed30 0x7efce80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.535+0000 7efd02a00700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efce806ed30 0x7efce80711e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efcf000b7a0 tx=0x7efcf0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:49.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.539+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7efcf405af70 con 0x7efd04103270 2024-04-16T11:45:49.668 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:49 smithi045 ceph-mon[81423]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:49.668 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:49 smithi045 ceph-mon[81423]: pgmap v37: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:49.668 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:49 smithi045 ceph-mon[81423]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:49.668 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:49 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2344008814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:49.668 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:49 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/971507004' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:45:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:49 smithi044 ceph-mon[93020]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:49 smithi044 ceph-mon[93020]: pgmap v37: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:49 smithi044 ceph-mon[93020]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:49 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2344008814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:49 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/971507004' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:45:49.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.784+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd04115dc0 con 0x7efce806ed30 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.786+0000 7efcfb400700 1 -- 172.21.15.44:0/3094141227 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7efd04115dc0 con 0x7efce806ed30 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-04-16T11:45:49.784 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:45:49.785 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "4/21 daemons upgraded", 2024-04-16T11:45:49.785 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T11:45:49.785 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:45:49.785 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:45:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efce806ed30 msgr2=0x7efce80711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efce806ed30 0x7efce80711e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efcf000b7a0 tx=0x7efcf0009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 msgr2=0x7efd04113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 0x7efd04113f20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efcf4002a50 tx=0x7efcf40046b0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 shutdown_connections 2024-04-16T11:45:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7efce806ed30 0x7efce80711e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd04103270 0x7efd04113f20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 --2- 172.21.15.44:0/3094141227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd04104520 0x7efd04114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 >> 172.21.15.44:0/3094141227 conn(0x7efd04072530 msgr2=0x7efd04074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.789+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 shutdown_connections 2024-04-16T11:45:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.790+0000 7efd09200700 1 -- 172.21.15.44:0/3094141227 wait complete. 2024-04-16T11:45:49.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 -- 172.21.15.44:0/989168868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fcdc0 msgr2=0x7fbc800fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 --2- 172.21.15.44:0/989168868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fcdc0 0x7fbc800fd230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c004940 tx=0x7fbc6c009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 -- 172.21.15.44:0/989168868 shutdown_connections 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 --2- 172.21.15.44:0/989168868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fcdc0 0x7fbc800fd230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 --2- 172.21.15.44:0/989168868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc800fbb10 0x7fbc800fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 -- 172.21.15.44:0/989168868 >> 172.21.15.44:0/989168868 conn(0x7fbc800f7270 msgr2=0x7fbc800f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 -- 172.21.15.44:0/989168868 shutdown_connections 2024-04-16T11:45:49.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.936+0000 7fbc86800700 1 -- 172.21.15.44:0/989168868 wait complete. 2024-04-16T11:45:49.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.937+0000 7fbc86800700 1 Processor -- start 2024-04-16T11:45:49.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc86800700 1 -- start start 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc86800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 0x7fbc8010fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc86800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc800fcdc0 0x7fbc8010ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc86800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc801105d0 con 0x7fbc800fbb10 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc86800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc80110e20 con 0x7fbc800fcdc0 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc85400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 0x7fbc8010fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc85400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 0x7fbc8010fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35042/0 (socket says 172.21.15.44:35042) 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.938+0000 7fbc85400700 1 -- 172.21.15.44:0/2304138726 learned_addr learned my addr 172.21.15.44:0/2304138726 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc84a00700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc800fcdc0 0x7fbc8010ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc85400700 1 -- 172.21.15.44:0/2304138726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc800fcdc0 msgr2=0x7fbc8010ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc85400700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc800fcdc0 0x7fbc8010ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc85400700 1 -- 172.21.15.44:0/2304138726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c005040 con 0x7fbc800fbb10 2024-04-16T11:45:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc85400700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 0x7fbc8010fa90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc74002a50 tx=0x7fbc740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc74007cc0 con 0x7fbc800fbb10 2024-04-16T11:45:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc80111100 con 0x7fbc800fbb10 2024-04-16T11:45:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc7400f040 con 0x7fbc800fbb10 2024-04-16T11:45:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.939+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc7400db70 con 0x7fbc800fbb10 2024-04-16T11:45:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.940+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc80111640 con 0x7fbc800fbb10 2024-04-16T11:45:49.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.941+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc80101750 con 0x7fbc800fbb10 2024-04-16T11:45:49.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.942+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 36) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fbc7400dd10 con 0x7fbc800fbb10 2024-04-16T11:45:49.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.943+0000 7fbc7d600700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbc6806ed20 0x7fbc680711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:45:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.943+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbc7408e270 con 0x7fbc800fbb10 2024-04-16T11:45:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.943+0000 7fbc84a00700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbc6806ed20 0x7fbc680711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:45:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.944+0000 7fbc84a00700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbc6806ed20 0x7fbc680711d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c004f90 tx=0x7fbc6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:45:49.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:49.948+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbc7405a8e0 con 0x7fbc800fbb10 2024-04-16T11:45:50.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.270+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbc8004bed0 con 0x7fbc800fbb10 2024-04-16T11:45:50.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.271+0000 7fbc7d600700 1 -- 172.21.15.44:0/2304138726 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fbc7405a470 con 0x7fbc800fbb10 2024-04-16T11:45:50.269 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:45:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbc6806ed20 msgr2=0x7fbc680711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbc6806ed20 0x7fbc680711d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c004f90 tx=0x7fbc6c009e30 comp rx=0 tx=0).stop 2024-04-16T11:45:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 msgr2=0x7fbc8010fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:45:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 0x7fbc8010fa90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc74002a50 tx=0x7fbc740046b0 comp rx=0 tx=0).stop 2024-04-16T11:45:50.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 shutdown_connections 2024-04-16T11:45:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbc6806ed20 0x7fbc680711d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbc800fbb10 0x7fbc8010fa90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 --2- 172.21.15.44:0/2304138726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc800fcdc0 0x7fbc8010ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:45:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 >> 172.21.15.44:0/2304138726 conn(0x7fbc800f7270 msgr2=0x7fbc80065b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:45:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.274+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 shutdown_connections 2024-04-16T11:45:50.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:45:50.275+0000 7fbc86800700 1 -- 172.21.15.44:0/2304138726 wait complete. 2024-04-16T11:45:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:50 smithi044 ceph-mon[93020]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T11:45:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:50.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:50 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2304138726' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:45:51.008 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:50 smithi045 ceph-mon[81423]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:45:51.008 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:51.008 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:51.008 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T11:45:51.009 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:51.009 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:50 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2304138726' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:45:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:51 smithi045 ceph-mon[81423]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T11:45:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:51 smithi045 ceph-mon[81423]: Reconfiguring daemon osd.5 on smithi045 2024-04-16T11:45:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:51 smithi045 ceph-mon[81423]: pgmap v38: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:51.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:51 smithi044 ceph-mon[93020]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T11:45:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:51 smithi044 ceph-mon[93020]: Reconfiguring daemon osd.5 on smithi045 2024-04-16T11:45:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:51 smithi044 ceph-mon[93020]: pgmap v38: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: pgmap v39: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: Reconfiguring mds.cephfs.smithi045.noofsd (monmap changed)... 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:53.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:53 smithi045 ceph-mon[81423]: Reconfiguring daemon mds.cephfs.smithi045.noofsd on smithi045 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: pgmap v39: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: Reconfiguring mds.cephfs.smithi045.noofsd (monmap changed)... 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:53.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:53 smithi044 ceph-mon[93020]: Reconfiguring daemon mds.cephfs.smithi045.noofsd on smithi045 2024-04-16T11:45:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:55 smithi045 ceph-mon[81423]: pgmap v40: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:55.877 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:55 smithi044 ceph-mon[93020]: pgmap v40: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:55.877 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:55.877 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:55.877 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:45:55.877 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:45:55.877 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:45:56.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:56 smithi045 ceph-mon[81423]: Reconfiguring mds.cephfs.smithi045.hxlkks (monmap changed)... 2024-04-16T11:45:56.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:56 smithi045 ceph-mon[81423]: Reconfiguring daemon mds.cephfs.smithi045.hxlkks on smithi045 2024-04-16T11:45:56.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:56 smithi044 ceph-mon[93020]: Reconfiguring mds.cephfs.smithi045.hxlkks (monmap changed)... 2024-04-16T11:45:56.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:56 smithi044 ceph-mon[93020]: Reconfiguring daemon mds.cephfs.smithi045.hxlkks on smithi045 2024-04-16T11:45:57.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: pgmap v41: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:57.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi044"}]: dispatch 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi044"}]': finished 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi045"}]: dispatch 2024-04-16T11:45:57.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi045"}]': finished 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: pgmap v41: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:45:57.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi044"}]: dispatch 2024-04-16T11:45:57.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi044"}]': finished 2024-04-16T11:45:57.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi045"}]: dispatch 2024-04-16T11:45:57.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi045"}]': finished 2024-04-16T11:45:58.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:58 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all mon 2024-04-16T11:45:58.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:58 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all mon 2024-04-16T11:45:59.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:45:59 smithi045 ceph-mon[81423]: pgmap v42: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:45:59.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:45:59 smithi044 ceph-mon[93020]: pgmap v42: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:00.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:00 smithi044 ceph-mon[93020]: Upgrade: Updating crash.smithi044 (1/2) 2024-04-16T11:46:00.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:00.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:46:00.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:00.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:00 smithi044 ceph-mon[93020]: Deploying daemon crash.smithi044 on smithi044 2024-04-16T11:46:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:00 smithi045 ceph-mon[81423]: Upgrade: Updating crash.smithi044 (1/2) 2024-04-16T11:46:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi044", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:46:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:00 smithi045 ceph-mon[81423]: Deploying daemon crash.smithi044 on smithi044 2024-04-16T11:46:01.933 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:01 smithi044 ceph-mon[93020]: pgmap v43: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:01 smithi045 ceph-mon[81423]: pgmap v43: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:04.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:03 smithi045 ceph-mon[81423]: pgmap v44: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:04.171 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:03 smithi044 ceph-mon[93020]: pgmap v44: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:06.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:05 smithi045 ceph-mon[81423]: pgmap v45: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:06.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:06.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:05 smithi044 ceph-mon[93020]: pgmap v45: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:08.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:07 smithi045 ceph-mon[81423]: pgmap v46: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:08.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:08.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:46:08.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:07 smithi044 ceph-mon[93020]: pgmap v46: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T11:46:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:08.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:08 smithi044 ceph-mon[93020]: Upgrade: Updating crash.smithi045 (2/2) 2024-04-16T11:46:08.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:08 smithi044 ceph-mon[93020]: Deploying daemon crash.smithi045 on smithi045 2024-04-16T11:46:09.020 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:08 smithi045 ceph-mon[81423]: Upgrade: Updating crash.smithi045 (2/2) 2024-04-16T11:46:09.020 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:08 smithi045 ceph-mon[81423]: Deploying daemon crash.smithi045 on smithi045 2024-04-16T11:46:10.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:09 smithi045 ceph-mon[81423]: pgmap v47: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:10.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:09 smithi044 ceph-mon[93020]: pgmap v47: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:11.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:12.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:11 smithi045 ceph-mon[81423]: pgmap v48: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:12.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:11 smithi044 ceph-mon[93020]: pgmap v48: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:13 smithi044 ceph-mon[93020]: pgmap v49: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:13 smithi044 ceph-mon[93020]: mgrmap e37: smithi044.sczobv(active, since 92s), standbys: smithi045.sdlwif 2024-04-16T11:46:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:13 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:13 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:13 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:46:14.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:13 smithi045 ceph-mon[81423]: pgmap v49: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:14.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:13 smithi045 ceph-mon[81423]: mgrmap e37: smithi044.sczobv(active, since 92s), standbys: smithi045.sdlwif 2024-04-16T11:46:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:13 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:13 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:13 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:46:16.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:15 smithi045 ceph-mon[81423]: pgmap v50: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:15 smithi044 ceph-mon[93020]: pgmap v50: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:17.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:17 smithi044 ceph-mon[93020]: pgmap v51: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:17.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:17.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:18.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:17 smithi045 ceph-mon[81423]: pgmap v51: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:18.348 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:18.349 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:20.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:19 smithi044 ceph-mon[93020]: pgmap v52: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:19 smithi045 ceph-mon[81423]: pgmap v52: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:20.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.435+0000 7fb5eac00700 1 -- 172.21.15.44:0/3621729094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 msgr2=0x7fb5e4072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:20.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 --2- 172.21.15.44:0/3621729094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4072c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4004940 tx=0x7fb5d4009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:20.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 -- 172.21.15.44:0/3621729094 shutdown_connections 2024-04-16T11:46:20.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 --2- 172.21.15.44:0/3621729094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e4073170 0x7fb5e4073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 --2- 172.21.15.44:0/3621729094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4072c30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 -- 172.21.15.44:0/3621729094 >> 172.21.15.44:0/3621729094 conn(0x7fb5e40fa950 msgr2=0x7fb5e40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 -- 172.21.15.44:0/3621729094 shutdown_connections 2024-04-16T11:46:20.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.436+0000 7fb5eac00700 1 -- 172.21.15.44:0/3621729094 wait complete. 2024-04-16T11:46:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.437+0000 7fb5eac00700 1 Processor -- start 2024-04-16T11:46:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.437+0000 7fb5eac00700 1 -- start start 2024-04-16T11:46:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5eac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5eac00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e41145c0 0x7fb5e4115410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5eac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4114a80 con 0x7fb5e4072820 2024-04-16T11:46:20.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5eac00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4114bc0 con 0x7fb5e41145c0 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41084/0 (socket says 172.21.15.44:41084) 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3e00700 1 -- 172.21.15.44:0/1472655225 learned_addr learned my addr 172.21.15.44:0/1472655225 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3400700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e41145c0 0x7fb5e4115410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3e00700 1 -- 172.21.15.44:0/1472655225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e41145c0 msgr2=0x7fb5e4115410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3e00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e41145c0 0x7fb5e4115410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.438+0000 7fb5e3e00700 1 -- 172.21.15.44:0/1472655225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d4005040 con 0x7fb5e4072820 2024-04-16T11:46:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.439+0000 7fb5e3e00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4114080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400b3b0 tx=0x7fb5d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.439+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5d400e040 con 0x7fb5e4072820 2024-04-16T11:46:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.439+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5d400bcc0 con 0x7fb5e4072820 2024-04-16T11:46:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.439+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e4115a10 con 0x7fb5e4072820 2024-04-16T11:46:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.439+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5d401c630 con 0x7fb5e4072820 2024-04-16T11:46:20.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.439+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e4115fa0 con 0x7fb5e4072820 2024-04-16T11:46:20.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.441+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e405f5f0 con 0x7fb5e4072820 2024-04-16T11:46:20.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.443+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb5d40139f0 con 0x7fb5e4072820 2024-04-16T11:46:20.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.444+0000 7fb5dbe00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb5c806ec00 0x7fb5c80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:20.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.444+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb5d408f530 con 0x7fb5e4072820 2024-04-16T11:46:20.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.444+0000 7fb5e3400700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb5c806ec00 0x7fb5c80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:20.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.445+0000 7fb5e3400700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb5c806ec00 0x7fb5c80710b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0004990 tx=0x7fb5d0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:20.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.450+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb5d405bee0 con 0x7fb5e4072820 2024-04-16T11:46:20.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.715+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5e4073170 con 0x7fb5c806ec00 2024-04-16T11:46:20.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.717+0000 7fb5dbe00700 1 -- 172.21.15.44:0/1472655225 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7fb5e4073170 con 0x7fb5c806ec00 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb5c806ec00 msgr2=0x7fb5c80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb5c806ec00 0x7fb5c80710b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0004990 tx=0x7fb5d0009ee0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 msgr2=0x7fb5e4114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4114080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400b3b0 tx=0x7fb5d4009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 shutdown_connections 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb5c806ec00 0x7fb5c80710b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb5e4072820 0x7fb5e4114080 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 --2- 172.21.15.44:0/1472655225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e41145c0 0x7fb5e4115410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.719+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 >> 172.21.15.44:0/1472655225 conn(0x7fb5e40fa950 msgr2=0x7fb5e40fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.720+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 shutdown_connections 2024-04-16T11:46:20.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.720+0000 7fb5eac00700 1 -- 172.21.15.44:0/1472655225 wait complete. 2024-04-16T11:46:20.732 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:46:20.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 -- 172.21.15.44:0/1086448879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07981002b0 msgr2=0x7f0798100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:20.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 --2- 172.21.15.44:0/1086448879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07981002b0 0x7f0798100720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0788004940 tx=0x7f0788009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:20.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 -- 172.21.15.44:0/1086448879 shutdown_connections 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 --2- 172.21.15.44:0/1086448879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07981002b0 0x7f0798100720 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 --2- 172.21.15.44:0/1086448879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07980ff000 0x7f07980ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 -- 172.21.15.44:0/1086448879 >> 172.21.15.44:0/1086448879 conn(0x7f07980fa950 msgr2=0x7f07980fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 -- 172.21.15.44:0/1086448879 shutdown_connections 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.879+0000 7f07a0800700 1 -- 172.21.15.44:0/1086448879 wait complete. 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.880+0000 7f07a0800700 1 Processor -- start 2024-04-16T11:46:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f07a0800700 1 -- start start 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f07a0800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 0x7f079810fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f07a0800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07981002b0 0x7f07981101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f07a0800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07981107d0 con 0x7f07980ff000 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f07a0800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0798110fd0 con 0x7f07981002b0 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f079de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 0x7f079810fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f079de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 0x7f079810fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41110/0 (socket says 172.21.15.44:41110) 2024-04-16T11:46:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.881+0000 7f079de00700 1 -- 172.21.15.44:0/2697403540 learned_addr learned my addr 172.21.15.44:0/2697403540 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.882+0000 7f079d400700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07981002b0 0x7f07981101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.882+0000 7f079de00700 1 -- 172.21.15.44:0/2697403540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07981002b0 msgr2=0x7f07981101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.882+0000 7f079de00700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07981002b0 0x7f07981101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.882+0000 7f079de00700 1 -- 172.21.15.44:0/2697403540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788005040 con 0x7f07980ff000 2024-04-16T11:46:20.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.882+0000 7f079d400700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07981002b0 0x7f07981101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:46:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.883+0000 7f079de00700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 0x7f079810fc90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f078c002a50 tx=0x7f078c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.883+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078c007cc0 con 0x7f07980ff000 2024-04-16T11:46:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.883+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f078c00f040 con 0x7f07980ff000 2024-04-16T11:46:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.883+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078c009640 con 0x7f07980ff000 2024-04-16T11:46:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.883+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07981112b0 con 0x7f07980ff000 2024-04-16T11:46:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.883+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07981117f0 con 0x7f07980ff000 2024-04-16T11:46:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.885+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079805f5f0 con 0x7f07980ff000 2024-04-16T11:46:20.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.886+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f078c0097c0 con 0x7f07980ff000 2024-04-16T11:46:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.887+0000 7f0792000700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f078006ec10 0x7f07800710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.888+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f078c08e5f0 con 0x7f07980ff000 2024-04-16T11:46:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.888+0000 7f079d400700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f078006ec10 0x7f07800710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:20.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.889+0000 7f079d400700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f078006ec10 0x7f07800710c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f078800b020 tx=0x7f0788009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:20.892+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f078c05af00 con 0x7f07980ff000 2024-04-16T11:46:20.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:20 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:20 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:20 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:20 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:21.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.151+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0798111be0 con 0x7f078006ec10 2024-04-16T11:46:21.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.153+0000 7f0792000700 1 -- 172.21.15.44:0/2697403540 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7f0798111be0 con 0x7f078006ec10 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.156+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f078006ec10 msgr2=0x7f07800710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.156+0000 7f07a0800700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f078006ec10 0x7f07800710c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f078800b020 tx=0x7f0788009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.156+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 msgr2=0x7f079810fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.156+0000 7f07a0800700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 0x7f079810fc90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f078c002a50 tx=0x7f078c0046b0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.156+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 shutdown_connections 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.156+0000 7f07a0800700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f078006ec10 0x7f07800710c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.157+0000 7f07a0800700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f07980ff000 0x7f079810fc90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.157+0000 7f07a0800700 1 --2- 172.21.15.44:0/2697403540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07981002b0 0x7f07981101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.157+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 >> 172.21.15.44:0/2697403540 conn(0x7f07980fa950 msgr2=0x7f07980fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:21.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.157+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 shutdown_connections 2024-04-16T11:46:21.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.157+0000 7f07a0800700 1 -- 172.21.15.44:0/2697403540 wait complete. 2024-04-16T11:46:21.204 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:20 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:21.205 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:20 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:21.205 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:20 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:21.205 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:20 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 -- 172.21.15.44:0/653421787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fce40 msgr2=0x7f77ec0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 --2- 172.21.15.44:0/653421787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fce40 0x7f77ec0fd250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f77dc004940 tx=0x7f77dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 -- 172.21.15.44:0/653421787 shutdown_connections 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 --2- 172.21.15.44:0/653421787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fe040 0x7f77ec0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 --2- 172.21.15.44:0/653421787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fce40 0x7f77ec0fd250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 -- 172.21.15.44:0/653421787 >> 172.21.15.44:0/653421787 conn(0x7f77ec0f8530 msgr2=0x7f77ec0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 -- 172.21.15.44:0/653421787 shutdown_connections 2024-04-16T11:46:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.319+0000 7f77f2200700 1 -- 172.21.15.44:0/653421787 wait complete. 2024-04-16T11:46:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.320+0000 7f77f2200700 1 Processor -- start 2024-04-16T11:46:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.320+0000 7f77f2200700 1 -- start start 2024-04-16T11:46:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.320+0000 7f77f2200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fce40 0x7f77ec10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.320+0000 7f77f2200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 0x7f77ec110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.320+0000 7f77f2200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec110850 con 0x7f77ec0fe040 2024-04-16T11:46:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.320+0000 7f77f2200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec110f90 con 0x7f77ec0fce40 2024-04-16T11:46:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 0x7f77ec110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 0x7f77ec110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41142/0 (socket says 172.21.15.44:41142) 2024-04-16T11:46:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eaa00700 1 -- 172.21.15.44:0/2828053139 learned_addr learned my addr 172.21.15.44:0/2828053139 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eb400700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fce40 0x7f77ec10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eaa00700 1 -- 172.21.15.44:0/2828053139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fce40 msgr2=0x7f77ec10fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eaa00700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fce40 0x7f77ec10fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eaa00700 1 -- 172.21.15.44:0/2828053139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc005040 con 0x7f77ec0fe040 2024-04-16T11:46:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.321+0000 7f77eb400700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fce40 0x7f77ec10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:46:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.322+0000 7f77eaa00700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 0x7f77ec110250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c830 tx=0x7f77d8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.322+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d800a760 con 0x7f77ec0fe040 2024-04-16T11:46:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.322+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77d800ad90 con 0x7f77ec0fe040 2024-04-16T11:46:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.322+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d80113a0 con 0x7f77ec0fe040 2024-04-16T11:46:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.322+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec111270 con 0x7f77ec0fe040 2024-04-16T11:46:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.322+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec1117e0 con 0x7f77ec0fe040 2024-04-16T11:46:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.323+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec1019d0 con 0x7f77ec0fe040 2024-04-16T11:46:21.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.325+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77d8012400 con 0x7f77ec0fe040 2024-04-16T11:46:21.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.326+0000 7f77e3400700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec10 0x7f77d00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:21.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.326+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f77d808e920 con 0x7f77ec0fe040 2024-04-16T11:46:21.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.326+0000 7f77eb400700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec10 0x7f77d00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:21.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.327+0000 7f77eb400700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec10 0x7f77d00710c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b3b0 tx=0x7f77dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:21.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.331+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f77d805b230 con 0x7f77ec0fe040 2024-04-16T11:46:21.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.577+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f77ec111b10 con 0x7f77d006ec10 2024-04-16T11:46:21.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.594+0000 7f77e3400700 1 -- 172.21.15.44:0/2828053139 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f77ec111b10 con 0x7f77d006ec10 2024-04-16T11:46:21.591 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:46:21.591 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (3m) 1s ago 13m 14.1M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:46:21.591 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (16s) 1s ago 14m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:46:21.591 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (8s) 3s ago 12m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (3m) 1s ago 13m 74.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (6m) 1s ago 6m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (6m) 1s ago 6m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (6m) 3s ago 6m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (6m) 3s ago 6m 13.4M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (3m) 1s ago 15m 497M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (2m) 3s ago 12m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (112s) 1s ago 15m 42.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (76s) 3s ago 12m 31.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (3m) 1s ago 14m 7818k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (3m) 3s ago 12m 8200k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (11m) 1s ago 11m 46.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 33aebd9fa376 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (10m) 1s ago 10m 49.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (10m) 1s ago 10m 46.9M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (9m) 3s ago 9m 46.4M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:46:21.592 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (9m) 3s ago 9m 44.8M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:46:21.593 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (8m) 3s ago 8m 46.6M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:46:21.593 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (2m) 1s ago 13m 53.4M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec10 msgr2=0x7f77d00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec10 0x7f77d00710c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b3b0 tx=0x7f77dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 msgr2=0x7f77ec110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 0x7f77ec110250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c830 tx=0x7f77d8004470 comp rx=0 tx=0).stop 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 shutdown_connections 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec10 0x7f77d00710c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec0fce40 0x7f77ec10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 --2- 172.21.15.44:0/2828053139 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec0fe040 0x7f77ec110250 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 >> 172.21.15.44:0/2828053139 conn(0x7f77ec0f8530 msgr2=0x7f77ec0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:21.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 shutdown_connections 2024-04-16T11:46:21.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.597+0000 7f77f2200700 1 -- 172.21.15.44:0/2828053139 wait complete. 2024-04-16T11:46:21.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.755+0000 7fdd7a400700 1 -- 172.21.15.44:0/3291926900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74100030 msgr2=0x7fdd74102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.755+0000 7fdd7a400700 1 --2- 172.21.15.44:0/3291926900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74100030 0x7fdd74102410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdd60004940 tx=0x7fdd60009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:21.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.755+0000 7fdd7a400700 1 -- 172.21.15.44:0/3291926900 shutdown_connections 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.755+0000 7fdd7a400700 1 --2- 172.21.15.44:0/3291926900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74100030 0x7fdd74102410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.755+0000 7fdd7a400700 1 --2- 172.21.15.44:0/3291926900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd74066230 0x7fdd740ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.755+0000 7fdd7a400700 1 -- 172.21.15.44:0/3291926900 >> 172.21.15.44:0/3291926900 conn(0x7fdd740f74c0 msgr2=0x7fdd740f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.756+0000 7fdd7a400700 1 -- 172.21.15.44:0/3291926900 shutdown_connections 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.756+0000 7fdd7a400700 1 -- 172.21.15.44:0/3291926900 wait complete. 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.756+0000 7fdd7a400700 1 Processor -- start 2024-04-16T11:46:21.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd7a400700 1 -- start start 2024-04-16T11:46:21.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd7a400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 0x7fdd7410da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:21.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd7a400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd74100030 0x7fdd7410df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:21.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd7a400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7410e590 con 0x7fdd74066230 2024-04-16T11:46:21.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd7a400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7410ede0 con 0x7fdd74100030 2024-04-16T11:46:21.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd73400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 0x7fdd7410da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:21.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd73400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 0x7fdd7410da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37686/0 (socket says 172.21.15.44:37686) 2024-04-16T11:46:21.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.757+0000 7fdd73400700 1 -- 172.21.15.44:0/2036101997 learned_addr learned my addr 172.21.15.44:0/2036101997 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:21.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.758+0000 7fdd72a00700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd74100030 0x7fdd7410df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.758+0000 7fdd73400700 1 -- 172.21.15.44:0/2036101997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd74100030 msgr2=0x7fdd7410df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.758+0000 7fdd73400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd74100030 0x7fdd7410df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.758+0000 7fdd73400700 1 -- 172.21.15.44:0/2036101997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60005040 con 0x7fdd74066230 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.758+0000 7fdd73400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 0x7fdd7410da50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd64002a80 tx=0x7fdd64004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.759+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd64007cc0 con 0x7fdd74066230 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.759+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd6400f040 con 0x7fdd74066230 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.759+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd6400dad0 con 0x7fdd74066230 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.759+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7410f0c0 con 0x7fdd74066230 2024-04-16T11:46:21.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.759+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7410f600 con 0x7fdd74066230 2024-04-16T11:46:21.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.760+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7405f5f0 con 0x7fdd74066230 2024-04-16T11:46:21.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.761+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd6400dc30 con 0x7fdd74066230 2024-04-16T11:46:21.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.762+0000 7fdd6b400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdd5806ec10 0x7fdd580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:21.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.762+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdd6408e380 con 0x7fdd74066230 2024-04-16T11:46:21.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.763+0000 7fdd72a00700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdd5806ec10 0x7fdd580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:21.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.764+0000 7fdd72a00700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdd5806ec10 0x7fdd580710c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000b3b0 tx=0x7fdd60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:21.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:21.768+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdd6405acb0 con 0x7fdd74066230 2024-04-16T11:46:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:21 smithi045 ceph-mon[81423]: pgmap v53: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:21 smithi045 ceph-mon[81423]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:22.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.088+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd7410f920 con 0x7fdd74066230 2024-04-16T11:46:22.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.090+0000 7fdd6b400700 1 -- 172.21.15.44:0/2036101997 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7fdd6405a840 con 0x7fdd74066230 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:46:22.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:22.088 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:46:22.088 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T11:46:22.088 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:46:22.088 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:46:22.088 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdd5806ec10 msgr2=0x7fdd580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdd5806ec10 0x7fdd580710c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000b3b0 tx=0x7fdd60009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 msgr2=0x7fdd7410da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 0x7fdd7410da50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd64002a80 tx=0x7fdd64004600 comp rx=0 tx=0).stop 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 shutdown_connections 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdd5806ec10 0x7fdd580710c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd74066230 0x7fdd7410da50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 --2- 172.21.15.44:0/2036101997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd74100030 0x7fdd7410df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 >> 172.21.15.44:0/2036101997 conn(0x7fdd740f74c0 msgr2=0x7fdd740f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:22.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.093+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 shutdown_connections 2024-04-16T11:46:22.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.094+0000 7fdd7a400700 1 -- 172.21.15.44:0/2036101997 wait complete. 2024-04-16T11:46:22.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:21 smithi044 ceph-mon[93020]: pgmap v53: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:22.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:21 smithi044 ceph-mon[93020]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.255+0000 7ff548800700 1 -- 172.21.15.44:0/1305638180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 msgr2=0x7ff5400fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.255+0000 7ff548800700 1 --2- 172.21.15.44:0/1305638180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff5400fd260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff534004940 tx=0x7ff534009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.255+0000 7ff548800700 1 -- 172.21.15.44:0/1305638180 shutdown_connections 2024-04-16T11:46:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.255+0000 7ff548800700 1 --2- 172.21.15.44:0/1305638180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5400fe050 0x7ff5400fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.255+0000 7ff548800700 1 --2- 172.21.15.44:0/1305638180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff5400fd260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.255+0000 7ff548800700 1 -- 172.21.15.44:0/1305638180 >> 172.21.15.44:0/1305638180 conn(0x7ff5400f8580 msgr2=0x7ff5400fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.256+0000 7ff548800700 1 -- 172.21.15.44:0/1305638180 shutdown_connections 2024-04-16T11:46:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.256+0000 7ff548800700 1 -- 172.21.15.44:0/1305638180 wait complete. 2024-04-16T11:46:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.256+0000 7ff548800700 1 Processor -- start 2024-04-16T11:46:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff548800700 1 -- start start 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff548800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff54010fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff548800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5400fe050 0x7ff540110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff548800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff540110f90 con 0x7ff5400fce50 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff548800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff540111100 con 0x7ff5400fe050 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff545e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff54010fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff545e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff54010fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37704/0 (socket says 172.21.15.44:37704) 2024-04-16T11:46:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff545e00700 1 -- 172.21.15.44:0/1817131088 learned_addr learned my addr 172.21.15.44:0/1817131088 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.257+0000 7ff545400700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5400fe050 0x7ff540110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff545e00700 1 -- 172.21.15.44:0/1817131088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5400fe050 msgr2=0x7ff540110280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff545e00700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5400fe050 0x7ff540110280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff545e00700 1 -- 172.21.15.44:0/1817131088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534005040 con 0x7ff5400fce50 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff545e00700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff54010fd40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff534000c00 tx=0x7ff534009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff53400e040 con 0x7ff5400fce50 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff53400bcc0 con 0x7ff5400fce50 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff53401c630 con 0x7ff5400fce50 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.258+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff540111380 con 0x7ff5400fce50 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.259+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540111940 con 0x7ff5400fce50 2024-04-16T11:46:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.260+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5401019e0 con 0x7ff5400fce50 2024-04-16T11:46:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.261+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff534013980 con 0x7ff5400fce50 2024-04-16T11:46:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.262+0000 7ff53a000700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff52806ec00 0x7ff5280710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:22.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.262+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff53408fb90 con 0x7ff5400fce50 2024-04-16T11:46:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.262+0000 7ff545400700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff52806ec00 0x7ff5280710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:22.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.263+0000 7ff545400700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff52806ec00 0x7ff5280710b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff530006fd0 tx=0x7ff530008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:22.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.267+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff53405c520 con 0x7ff5400fce50 2024-04-16T11:46:22.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.585+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff540111dd0 con 0x7ff5400fce50 2024-04-16T11:46:22.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.586+0000 7ff53a000700 1 -- 172.21.15.44:0/1817131088 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7ff53405c0b0 con 0x7ff5400fce50 2024-04-16T11:46:22.583 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:46:22.583 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:46:22.583 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:46:22.584 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:46:22.585 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:22.586 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff52806ec00 msgr2=0x7ff5280710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff52806ec00 0x7ff5280710b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff530006fd0 tx=0x7ff530008040 comp rx=0 tx=0).stop 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 msgr2=0x7ff54010fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff54010fd40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff534000c00 tx=0x7ff534009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 shutdown_connections 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff52806ec00 0x7ff5280710b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5400fce50 0x7ff54010fd40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 --2- 172.21.15.44:0/1817131088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5400fe050 0x7ff540110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 >> 172.21.15.44:0/1817131088 conn(0x7ff5400f8580 msgr2=0x7ff5400f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 shutdown_connections 2024-04-16T11:46:22.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.589+0000 7ff548800700 1 -- 172.21.15.44:0/1817131088 wait complete. 2024-04-16T11:46:22.588 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 -- 172.21.15.44:0/4212216264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce04071ea0 msgr2=0x7fce040703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 --2- 172.21.15.44:0/4212216264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce04071ea0 0x7fce040703b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4004940 tx=0x7fcdf4009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 -- 172.21.15.44:0/4212216264 shutdown_connections 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 --2- 172.21.15.44:0/4212216264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce040708f0 0x7fce04070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 --2- 172.21.15.44:0/4212216264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce04071ea0 0x7fce040703b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 -- 172.21.15.44:0/4212216264 >> 172.21.15.44:0/4212216264 conn(0x7fce040f83d0 msgr2=0x7fce040fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 -- 172.21.15.44:0/4212216264 shutdown_connections 2024-04-16T11:46:22.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.744+0000 7fce09400700 1 -- 172.21.15.44:0/4212216264 wait complete. 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce09400700 1 Processor -- start 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce09400700 1 -- start start 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce09400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 0x7fce04113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce09400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce04071ea0 0x7fce04114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce09400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce041151d0 con 0x7fce040708f0 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce09400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04115340 con 0x7fce04071ea0 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce03e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 0x7fce04113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce03e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 0x7fce04113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37722/0 (socket says 172.21.15.44:37722) 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.745+0000 7fce03e00700 1 -- 172.21.15.44:0/2202549188 learned_addr learned my addr 172.21.15.44:0/2202549188 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fce03400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce04071ea0 0x7fce04114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fce03e00700 1 -- 172.21.15.44:0/2202549188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce04071ea0 msgr2=0x7fce04114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fce03e00700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce04071ea0 0x7fce04114460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:22.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fce03e00700 1 -- 172.21.15.44:0/2202549188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4005040 con 0x7fce040708f0 2024-04-16T11:46:22.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fce03e00700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 0x7fce04113f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400b970 tx=0x7fcdf4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:22.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdf400e040 con 0x7fce040708f0 2024-04-16T11:46:22.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdf400bbf0 con 0x7fce040708f0 2024-04-16T11:46:22.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdf401c630 con 0x7fce040708f0 2024-04-16T11:46:22.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.746+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce04115560 con 0x7fce040708f0 2024-04-16T11:46:22.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.747+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce04115b20 con 0x7fce040708f0 2024-04-16T11:46:22.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.748+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0410c540 con 0x7fce040708f0 2024-04-16T11:46:22.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.749+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcdf400bd60 con 0x7fce040708f0 2024-04-16T11:46:22.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.750+0000 7fcdfbe00700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcde806ec10 0x7fcde80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:22.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.750+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcdf408fbd0 con 0x7fce040708f0 2024-04-16T11:46:22.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.751+0000 7fce03400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcde806ec10 0x7fcde80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:22.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.751+0000 7fce03400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcde806ec10 0x7fcde80710c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0004990 tx=0x7fcdf0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:22.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:22.755+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcdf4058a30 con 0x7fce040708f0 2024-04-16T11:46:23.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.005+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce04115e50 con 0x7fcde806ec10 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.008+0000 7fcdfbe00700 1 -- 172.21.15.44:0/2202549188 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7fce04115e50 con 0x7fcde806ec10 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:46:23.005 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:46:23.006 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "6/21 daemons upgraded", 2024-04-16T11:46:23.006 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading crash daemons", 2024-04-16T11:46:23.006 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:46:23.006 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.011+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcde806ec10 msgr2=0x7fcde80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.011+0000 7fce09400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcde806ec10 0x7fcde80710c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0004990 tx=0x7fcdf0009ee0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.011+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 msgr2=0x7fce04113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.011+0000 7fce09400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 0x7fce04113f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400b970 tx=0x7fcdf4009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 shutdown_connections 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcde806ec10 0x7fcde80710c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce040708f0 0x7fce04113f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 --2- 172.21.15.44:0/2202549188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce04071ea0 0x7fce04114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 >> 172.21.15.44:0/2202549188 conn(0x7fce040f83d0 msgr2=0x7fce040f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 shutdown_connections 2024-04-16T11:46:23.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.012+0000 7fce09400700 1 -- 172.21.15.44:0/2202549188 wait complete. 2024-04-16T11:46:23.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.174+0000 7f16bf200700 1 -- 172.21.15.44:0/1873875325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fcef0 msgr2=0x7f16b80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:23.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.174+0000 7f16bf200700 1 --2- 172.21.15.44:0/1873875325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fcef0 0x7f16b80fd320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004940 tx=0x7f16a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:23.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.174+0000 7f16bf200700 1 -- 172.21.15.44:0/1873875325 shutdown_connections 2024-04-16T11:46:23.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.174+0000 7f16bf200700 1 --2- 172.21.15.44:0/1873875325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16b80fe0e0 0x7f16b80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.174+0000 7f16bf200700 1 --2- 172.21.15.44:0/1873875325 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fcef0 0x7f16b80fd320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.174+0000 7f16bf200700 1 -- 172.21.15.44:0/1873875325 >> 172.21.15.44:0/1873875325 conn(0x7f16b80f8530 msgr2=0x7f16b80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:23.172 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:22 smithi044 ceph-mon[93020]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:23.172 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:22 smithi044 ceph-mon[93020]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:23.172 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:22 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2036101997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:23.172 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:22 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:23.172 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:22 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:23.172 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:22 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1817131088' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:46:23.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.175+0000 7f16bf200700 1 -- 172.21.15.44:0/1873875325 shutdown_connections 2024-04-16T11:46:23.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.175+0000 7f16bf200700 1 -- 172.21.15.44:0/1873875325 wait complete. 2024-04-16T11:46:23.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.176+0000 7f16bf200700 1 Processor -- start 2024-04-16T11:46:23.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.176+0000 7f16bf200700 1 -- start start 2024-04-16T11:46:23.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.176+0000 7f16bf200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16b80fcef0 0x7f16b810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:23.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.176+0000 7f16bf200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 0x7f16b8110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:23.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.176+0000 7f16bf200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8110810 con 0x7f16b80fe0e0 2024-04-16T11:46:23.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.176+0000 7f16bf200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8111000 con 0x7f16b80fcef0 2024-04-16T11:46:23.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 0x7f16b8110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 0x7f16b8110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37738/0 (socket says 172.21.15.44:37738) 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7400700 1 -- 172.21.15.44:0/3343157557 learned_addr learned my addr 172.21.15.44:0/3343157557 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7e00700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16b80fcef0 0x7f16b810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7400700 1 -- 172.21.15.44:0/3343157557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16b80fcef0 msgr2=0x7f16b810fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7400700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16b80fcef0 0x7f16b810fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.177+0000 7f16b7400700 1 -- 172.21.15.44:0/3343157557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a8005040 con 0x7f16b80fe0e0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.178+0000 7f16b7400700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 0x7f16b8110180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f16a400b790 tx=0x7f16a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.178+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16a400f6f0 con 0x7f16b80fe0e0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.178+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16a400fd20 con 0x7f16b80fe0e0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.178+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b81112e0 con 0x7f16b80fe0e0 2024-04-16T11:46:23.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.178+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16a40103a0 con 0x7f16b80fe0e0 2024-04-16T11:46:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.178+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b81118a0 con 0x7f16b80fe0e0 2024-04-16T11:46:23.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.180+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b8101a70 con 0x7f16b80fe0e0 2024-04-16T11:46:23.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.183+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16a4011400 con 0x7f16b80fe0e0 2024-04-16T11:46:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.183+0000 7f16afe00700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f169c06ec10 0x7f169c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:23.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.184+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f16a408e660 con 0x7f16b80fe0e0 2024-04-16T11:46:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.184+0000 7f16b7e00700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f169c06ec10 0x7f169c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:23.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.185+0000 7f16b7e00700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f169c06ec10 0x7f169c0710c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16a800b3b0 tx=0x7f16a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:23.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.190+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f16a405b010 con 0x7f16b80fe0e0 2024-04-16T11:46:23.202 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:22 smithi045 ceph-mon[81423]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:23.202 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:22 smithi045 ceph-mon[81423]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:23.202 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:22 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2036101997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:23.202 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:22 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:23.202 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:22 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:23.202 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:22 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1817131088' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:46:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.549+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f16b805f5f0 con 0x7f16b80fe0e0 2024-04-16T11:46:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.550+0000 7f16afe00700 1 -- 172.21.15.44:0/3343157557 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f16a405aba0 con 0x7f16b80fe0e0 2024-04-16T11:46:23.547 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:46:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f169c06ec10 msgr2=0x7f169c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f169c06ec10 0x7f169c0710c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16a800b3b0 tx=0x7f16a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 msgr2=0x7f16b8110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 0x7f16b8110180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f16a400b790 tx=0x7f16a4009250 comp rx=0 tx=0).stop 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 shutdown_connections 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f169c06ec10 0x7f169c0710c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16b80fcef0 0x7f16b810fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 --2- 172.21.15.44:0/3343157557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16b80fe0e0 0x7f16b8110180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 >> 172.21.15.44:0/3343157557 conn(0x7f16b80f8530 msgr2=0x7f16b80f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 shutdown_connections 2024-04-16T11:46:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:23.553+0000 7f16bf200700 1 -- 172.21.15.44:0/3343157557 wait complete. 2024-04-16T11:46:23.774 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 292, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86990 2024-04-16T11:46:23.775 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 292, success, outsize: 120 2024-04-16T11:46:23.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 294, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87083 2024-04-16T11:46:23.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 294, success, outsize: 120 2024-04-16T11:46:23.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 296, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87179 2024-04-16T11:46:23.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 296, success, outsize: 120 2024-04-16T11:46:23.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 298, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87281 2024-04-16T11:46:23.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 298, success, outsize: 120 2024-04-16T11:46:23.893 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 300, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87380 2024-04-16T11:46:23.893 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 300, success, outsize: 120 2024-04-16T11:46:23.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 302, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87473 2024-04-16T11:46:23.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 302, success, outsize: 120 2024-04-16T11:46:23.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 304, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87569 2024-04-16T11:46:23.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 304, success, outsize: 120 2024-04-16T11:46:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 306, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87671 2024-04-16T11:46:23.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 306, success, outsize: 120 2024-04-16T11:46:24.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 308, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87770 2024-04-16T11:46:24.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 308, success, outsize: 120 2024-04-16T11:46:24.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 310, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87863 2024-04-16T11:46:24.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 310, success, outsize: 120 2024-04-16T11:46:24.057 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:23 smithi044 ceph-mon[93020]: pgmap v54: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:24.057 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:23 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3343157557' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:46:24.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 312, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87959 2024-04-16T11:46:24.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 312, success, outsize: 120 2024-04-16T11:46:24.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 314, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88061 2024-04-16T11:46:24.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 314, success, outsize: 120 2024-04-16T11:46:24.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 316, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88160 2024-04-16T11:46:24.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 316, success, outsize: 120 2024-04-16T11:46:24.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 318, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88253 2024-04-16T11:46:24.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 318, success, outsize: 120 2024-04-16T11:46:24.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 320, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88349 2024-04-16T11:46:24.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 320, success, outsize: 120 2024-04-16T11:46:24.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 322, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88451 2024-04-16T11:46:24.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 322, success, outsize: 120 2024-04-16T11:46:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 324, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88550 2024-04-16T11:46:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 324, success, outsize: 120 2024-04-16T11:46:24.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 326, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88643 2024-04-16T11:46:24.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 326, success, outsize: 120 2024-04-16T11:46:24.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 328, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88739 2024-04-16T11:46:24.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 328, success, outsize: 120 2024-04-16T11:46:24.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:23 smithi045 ceph-mon[81423]: pgmap v54: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T11:46:24.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:23 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3343157557' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:46:24.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 330, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88841 2024-04-16T11:46:24.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 330, success, outsize: 120 2024-04-16T11:46:24.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 332, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88940 2024-04-16T11:46:24.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 332, success, outsize: 120 2024-04-16T11:46:24.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 334, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89033 2024-04-16T11:46:24.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 334, success, outsize: 120 2024-04-16T11:46:24.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 336, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89129 2024-04-16T11:46:24.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 336, success, outsize: 120 2024-04-16T11:46:24.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 338, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89231 2024-04-16T11:46:24.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 338, success, outsize: 120 2024-04-16T11:46:24.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 340, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89330 2024-04-16T11:46:24.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 340, success, outsize: 120 2024-04-16T11:46:24.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 342, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89423 2024-04-16T11:46:24.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 342, success, outsize: 120 2024-04-16T11:46:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 344, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89519 2024-04-16T11:46:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 344, success, outsize: 120 2024-04-16T11:46:24.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 346, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89621 2024-04-16T11:46:24.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 346, success, outsize: 120 2024-04-16T11:46:25.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:24 smithi044 ceph-mon[93020]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:25.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:24 smithi045 ceph-mon[81423]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:26.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:25 smithi044 ceph-mon[93020]: pgmap v55: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:26.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:26.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:26.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:25 smithi045 ceph-mon[81423]: pgmap v55: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:26.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:26.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 124, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98354 2024-04-16T11:46:26.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 124, success, outsize: 120 2024-04-16T11:46:26.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 126, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98452 2024-04-16T11:46:26.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 126, success, outsize: 120 2024-04-16T11:46:26.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 128, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98553 2024-04-16T11:46:26.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 128, success, outsize: 120 2024-04-16T11:46:26.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 130, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98660 2024-04-16T11:46:26.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 130, success, outsize: 120 2024-04-16T11:46:26.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 132, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98764 2024-04-16T11:46:26.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 132, success, outsize: 120 2024-04-16T11:46:26.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 134, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98862 2024-04-16T11:46:26.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 134, success, outsize: 120 2024-04-16T11:46:26.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 136, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98963 2024-04-16T11:46:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 136, success, outsize: 120 2024-04-16T11:46:26.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 138, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99070 2024-04-16T11:46:26.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 138, success, outsize: 120 2024-04-16T11:46:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 140, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99174 2024-04-16T11:46:26.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 140, success, outsize: 120 2024-04-16T11:46:27.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 142, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99272 2024-04-16T11:46:27.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 142, success, outsize: 120 2024-04-16T11:46:27.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 144, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99373 2024-04-16T11:46:27.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 144, success, outsize: 120 2024-04-16T11:46:27.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 146, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99480 2024-04-16T11:46:27.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 146, success, outsize: 120 2024-04-16T11:46:27.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 148, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99584 2024-04-16T11:46:27.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 148, success, outsize: 120 2024-04-16T11:46:27.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 150, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99682 2024-04-16T11:46:27.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 150, success, outsize: 120 2024-04-16T11:46:27.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 152, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99783 2024-04-16T11:46:27.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 152, success, outsize: 120 2024-04-16T11:46:27.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 154, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99890 2024-04-16T11:46:27.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 154, success, outsize: 120 2024-04-16T11:46:27.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 156, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99994 2024-04-16T11:46:27.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 156, success, outsize: 120 2024-04-16T11:46:27.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:26 smithi044 ceph-mon[93020]: pgmap v56: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 158, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100092 2024-04-16T11:46:27.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 158, success, outsize: 120 2024-04-16T11:46:27.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 160, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100193 2024-04-16T11:46:27.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 160, success, outsize: 120 2024-04-16T11:46:27.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 162, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100300 2024-04-16T11:46:27.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 162, success, outsize: 120 2024-04-16T11:46:27.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 164, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100404 2024-04-16T11:46:27.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 164, success, outsize: 120 2024-04-16T11:46:27.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:26 smithi045 ceph-mon[81423]: pgmap v56: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:27.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 166, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100502 2024-04-16T11:46:27.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 166, success, outsize: 120 2024-04-16T11:46:27.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 168, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100603 2024-04-16T11:46:27.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 168, success, outsize: 120 2024-04-16T11:46:27.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 170, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100710 2024-04-16T11:46:27.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 170, success, outsize: 120 2024-04-16T11:46:27.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 172, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100814 2024-04-16T11:46:27.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 172, success, outsize: 120 2024-04-16T11:46:27.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 174, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100912 2024-04-16T11:46:27.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 174, success, outsize: 120 2024-04-16T11:46:27.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 176, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101013 2024-04-16T11:46:27.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 176, success, outsize: 120 2024-04-16T11:46:27.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 178, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101120 2024-04-16T11:46:27.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 178, success, outsize: 120 2024-04-16T11:46:29.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: pgmap v57: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all crash 2024-04-16T11:46:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi044"}]: dispatch 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi044"}]': finished 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi045"}]: dispatch 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi045"}]': finished 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T11:46:29.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:29 smithi044 ceph-mon[93020]: Upgrade: osd.0 is safe to restart 2024-04-16T11:46:29.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: pgmap v57: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:29.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:29.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all crash 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi044"}]: dispatch 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi044"}]': finished 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi045"}]: dispatch 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi045"}]': finished 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T11:46:29.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:29 smithi045 ceph-mon[81423]: Upgrade: osd.0 is safe to restart 2024-04-16T11:46:31.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:31 smithi044 ceph-mon[93020]: pgmap v58: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:31.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:31 smithi045 ceph-mon[81423]: pgmap v58: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:32 smithi044 ceph-mon[93020]: Upgrade: Updating osd.0 2024-04-16T11:46:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:32 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:32 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T11:46:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:32 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:32 smithi044 ceph-mon[93020]: Deploying daemon osd.0 on smithi044 2024-04-16T11:46:32.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:32 smithi045 ceph-mon[81423]: Upgrade: Updating osd.0 2024-04-16T11:46:32.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:32 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:32.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:32 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T11:46:32.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:32 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:46:32.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:32 smithi045 ceph-mon[81423]: Deploying daemon osd.0 on smithi044 2024-04-16T11:46:33.390 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:33 smithi044 ceph-mon[93020]: pgmap v59: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:33.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:33 smithi045 ceph-mon[81423]: pgmap v59: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:35.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:35 smithi044 ceph-mon[93020]: pgmap v60: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:35.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:35 smithi045 ceph-mon[81423]: pgmap v60: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:36.476 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:36 smithi044 systemd[1]: Stopping Ceph osd.0 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:46:36.976 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:36 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[47461]: 2024-04-16T11:46:36.703+0000 7f6bff400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:46:36.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:36 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[47461]: 2024-04-16T11:46:36.703+0000 7f6bff400700 -1 osd.0 48 *** Got signal Terminated *** 2024-04-16T11:46:36.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:36 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[47461]: 2024-04-16T11:46:36.703+0000 7f6bff400700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T11:46:37.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:37 smithi044 ceph-mon[93020]: pgmap v61: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:37.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:37 smithi044 ceph-mon[93020]: osd.0 marked itself down and dead 2024-04-16T11:46:37.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:37 smithi045 ceph-mon[81423]: pgmap v61: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:37.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:37 smithi045 ceph-mon[81423]: osd.0 marked itself down and dead 2024-04-16T11:46:37.860 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:37 smithi044 podman[101576]: 2024-04-16 11:46:37.569990099 +0000 UTC m=+1.008631147 container died 33aebd9fa376b2c190fb51dca4033e7af323538687f2180551eb8a2bf115a6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-16T11:46:38.206 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:38 smithi044 podman[101576]: 2024-04-16 11:46:38.036872879 +0000 UTC m=+1.475513968 container cleanup 33aebd9fa376b2c190fb51dca4033e7af323538687f2180551eb8a2bf115a6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-16T11:46:38.207 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:38 smithi044 bash[101576]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0 2024-04-16T11:46:38.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:38 smithi044 ceph-mon[93020]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:46:38.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:38 smithi044 ceph-mon[93020]: osdmap e49: 6 total, 5 up, 6 in 2024-04-16T11:46:38.476 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:38 smithi044 podman[101593]: 2024-04-16 11:46:38.288176542 +0000 UTC m=+0.695379607 container remove 33aebd9fa376b2c190fb51dca4033e7af323538687f2180551eb8a2bf115a6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-16T11:46:38.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:38 smithi045 ceph-mon[81423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:46:38.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:38 smithi045 ceph-mon[81423]: osdmap e49: 6 total, 5 up, 6 in 2024-04-16T11:46:39.381 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:39 smithi044 ceph-mon[93020]: pgmap v63: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:39.381 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:39 smithi044 ceph-mon[93020]: osdmap e50: 6 total, 5 up, 6 in 2024-04-16T11:46:39.381 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:39 smithi044 podman[101685]: 2024-04-16 11:46:39.04040449 +0000 UTC m=+0.042512189 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:46:39.381 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:39 smithi044 podman[101685]: 2024-04-16 11:46:39.155680906 +0000 UTC m=+0.157788665 container create 152296a4c152ae136bdf7ba06b61f5dc782da40de1fcd22092647316ae2cc1b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-16T11:46:39.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:39 smithi045 ceph-mon[81423]: pgmap v63: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:39.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:39 smithi045 ceph-mon[81423]: osdmap e50: 6 total, 5 up, 6 in 2024-04-16T11:46:39.976 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:39 smithi044 podman[101685]: 2024-04-16 11:46:39.501681179 +0000 UTC m=+0.503788917 container init 152296a4c152ae136bdf7ba06b61f5dc782da40de1fcd22092647316ae2cc1b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-16T11:46:39.976 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:39 smithi044 podman[101685]: 2024-04-16 11:46:39.514023334 +0000 UTC m=+0.516131038 container start 152296a4c152ae136bdf7ba06b61f5dc782da40de1fcd22092647316ae2cc1b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T11:46:39.976 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:39 smithi044 podman[101685]: 2024-04-16 11:46:39.616629259 +0000 UTC m=+0.618737041 container attach 152296a4c152ae136bdf7ba06b61f5dc782da40de1fcd22092647316ae2cc1b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:46:40.243 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:39 smithi044 podman[101685]: 2024-04-16 11:46:39.989183691 +0000 UTC m=+0.991291433 container died 152296a4c152ae136bdf7ba06b61f5dc782da40de1fcd22092647316ae2cc1b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True) 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [3, 0]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 2]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [3, 0]}]: dispatch 2024-04-16T11:46:40.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0, 1, 2]}]: dispatch 2024-04-16T11:46:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2]}]: dispatch 2024-04-16T11:46:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]: dispatch 2024-04-16T11:46:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [3, 0]}]: dispatch 2024-04-16T11:46:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]: dispatch 2024-04-16T11:46:40.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 2]}]: dispatch 2024-04-16T11:46:40.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]: dispatch 2024-04-16T11:46:40.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [3, 0]}]: dispatch 2024-04-16T11:46:40.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0, 1, 2]}]: dispatch 2024-04-16T11:46:41.212 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:40 smithi044 podman[101711]: 2024-04-16 11:46:40.755632256 +0000 UTC m=+0.743996010 container remove 152296a4c152ae136bdf7ba06b61f5dc782da40de1fcd22092647316ae2cc1b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-16T11:46:41.212 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:40 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-16T11:46:41.212 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:40 smithi044 systemd[1]: Stopped Ceph osd.0 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:46:41.212 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:40 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.0.service: Consumed 12.642s CPU time. 2024-04-16T11:46:41.212 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:41 smithi044 systemd[1]: Starting Ceph osd.0 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:46:41.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: pgmap v65: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:41.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2]}]': finished 2024-04-16T11:46:41.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]': finished 2024-04-16T11:46:41.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [3, 0]}]': finished 2024-04-16T11:46:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]': finished 2024-04-16T11:46:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 2]}]': finished 2024-04-16T11:46:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]': finished 2024-04-16T11:46:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [3, 0]}]': finished 2024-04-16T11:46:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0, 1, 2]}]': finished 2024-04-16T11:46:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T11:46:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: pgmap v65: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [3, 0]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 2]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 5]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [3, 0]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0, 1, 2]}]': finished 2024-04-16T11:46:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T11:46:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:41.881 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:41 smithi044 podman[101812]: 2024-04-16 11:46:41.490193821 +0000 UTC m=+0.034580022 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:46:41.881 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:41 smithi044 podman[101812]: 2024-04-16 11:46:41.596418328 +0000 UTC m=+0.140804577 container create b6c29faf74d242c8e343e5cf4c9e206e941533decbae9c5a4e3b26586e85cda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T11:46:41.881 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:41 smithi044 podman[101812]: 2024-04-16 11:46:41.883492125 +0000 UTC m=+0.427878370 container init b6c29faf74d242c8e343e5cf4c9e206e941533decbae9c5a4e3b26586e85cda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T11:46:42.226 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:41 smithi044 podman[101812]: 2024-04-16 11:46:41.897020131 +0000 UTC m=+0.441406353 container start b6c29faf74d242c8e343e5cf4c9e206e941533decbae9c5a4e3b26586e85cda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True) 2024-04-16T11:46:42.226 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 podman[101812]: 2024-04-16 11:46:42.007395071 +0000 UTC m=+0.551781316 container attach b6c29faf74d242c8e343e5cf4c9e206e941533decbae9c5a4e3b26586e85cda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T11:46:42.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:42 smithi045 ceph-mon[81423]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T11:46:42.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 180, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101906 2024-04-16T11:46:42.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 180, success, outsize: 120 2024-04-16T11:46:42.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-mon[93020]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T11:46:43.227 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T11:46:43.227 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 bash[101812]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate[101827]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T11:46:43.228 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:42 smithi044 podman[101812]: 2024-04-16 11:46:42.877141867 +0000 UTC m=+1.421528092 container died b6c29faf74d242c8e343e5cf4c9e206e941533decbae9c5a4e3b26586e85cda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:46:43.506 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:43 smithi044 ceph-mon[93020]: pgmap v68: 65 pgs: 6 peering, 8 active+undersized+degraded, 22 active+undersized, 29 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T11:46:43.506 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:43 smithi044 ceph-mon[93020]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-16T11:46:43.506 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:43 smithi044 ceph-mon[93020]: Health check failed: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T11:46:43.506 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:43 smithi044 ceph-mon[93020]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T11:46:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:43 smithi045 ceph-mon[81423]: pgmap v68: 65 pgs: 6 peering, 8 active+undersized+degraded, 22 active+undersized, 29 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T11:46:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:43 smithi045 ceph-mon[81423]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-16T11:46:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:43 smithi045 ceph-mon[81423]: Health check failed: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T11:46:43.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:43 smithi045 ceph-mon[81423]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T11:46:43.975 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:43 smithi044 podman[101950]: 2024-04-16 11:46:43.610893304 +0000 UTC m=+0.711069257 container remove b6c29faf74d242c8e343e5cf4c9e206e941533decbae9c5a4e3b26586e85cda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T11:46:44.226 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:43 smithi044 podman[102008]: 2024-04-16 11:46:43.887902212 +0000 UTC m=+0.042287934 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:46:44.226 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 podman[102008]: 2024-04-16 11:46:44.020182499 +0000 UTC m=+0.174568246 container create 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS) 2024-04-16T11:46:44.726 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 podman[102008]: 2024-04-16 11:46:44.298958955 +0000 UTC m=+0.453344702 container init 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-16T11:46:44.726 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 podman[102008]: 2024-04-16 11:46:44.311139022 +0000 UTC m=+0.465524792 container start 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-16T11:46:44.726 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 ceph-osd[102027]: -- 172.21.15.44:0/2876024167 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d36d2e1680 con 0x55d36d2ab000 2024-04-16T11:46:44.726 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 bash[102008]: 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 2024-04-16T11:46:44.726 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 systemd[1]: Started Ceph osd.0 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:46:45.176 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[102023]: 2024-04-16T11:46:44.892+0000 7f6b9128e7c0 -1 Falling back to public interface 2024-04-16T11:46:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:45 smithi045 ceph-mon[81423]: pgmap v70: 65 pgs: 6 peering, 8 active+undersized+degraded, 22 active+undersized, 29 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T11:46:45.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:45.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:45.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:46:45.920 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:45 smithi044 ceph-mon[93020]: pgmap v70: 65 pgs: 6 peering, 8 active+undersized+degraded, 22 active+undersized, 29 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-04-16T11:46:45.920 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:45.920 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:45.920 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:46:46.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:46 smithi044 ceph-mon[93020]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-16T11:46:47.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:46 smithi045 ceph-mon[81423]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-16T11:46:47.959 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:47 smithi044 ceph-mon[93020]: pgmap v71: 65 pgs: 8 active+clean+remapped, 8 active+undersized+degraded, 22 active+undersized, 27 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%); 4/72 objects misplaced (5.556%) 2024-04-16T11:46:48.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:47 smithi045 ceph-mon[81423]: pgmap v71: 65 pgs: 8 active+clean+remapped, 8 active+undersized+degraded, 22 active+undersized, 27 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%); 4/72 objects misplaced (5.556%) 2024-04-16T11:46:48.897 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:48 smithi044 ceph-mon[93020]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T11:46:48.898 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:48 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[102023]: 2024-04-16T11:46:48.555+0000 7f6b9128e7c0 -1 osd.0 48 log_to_monitors true 2024-04-16T11:46:49.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:48 smithi045 ceph-mon[81423]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T11:46:49.934 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 11:46:49 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[102023]: 2024-04-16T11:46:49.696+0000 7f6b7b600700 -1 osd.0 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:46:49.935 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:49 smithi044 ceph-mon[93020]: pgmap v72: 65 pgs: 8 active+clean+remapped, 8 active+undersized+degraded, 22 active+undersized, 27 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%); 4/72 objects misplaced (5.556%) 2024-04-16T11:46:49.935 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:49 smithi044 ceph-mon[93020]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T11:46:49.935 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:49 smithi044 ceph-mon[93020]: osdmap e54: 6 total, 5 up, 6 in 2024-04-16T11:46:49.935 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:49 smithi044 ceph-mon[93020]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:46:50.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:49 smithi045 ceph-mon[81423]: pgmap v72: 65 pgs: 8 active+clean+remapped, 8 active+undersized+degraded, 22 active+undersized, 27 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%); 4/72 objects misplaced (5.556%) 2024-04-16T11:46:50.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:49 smithi045 ceph-mon[81423]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T11:46:50.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:49 smithi045 ceph-mon[81423]: osdmap e54: 6 total, 5 up, 6 in 2024-04-16T11:46:50.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:49 smithi045 ceph-mon[81423]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:46:50.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:50 smithi044 ceph-mon[93020]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' 2024-04-16T11:46:51.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:50 smithi045 ceph-mon[81423]: from='osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871]' entity='osd.0' 2024-04-16T11:46:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:51 smithi044 ceph-mon[93020]: pgmap v74: 65 pgs: 8 active+clean+remapped, 8 active+undersized+degraded, 22 active+undersized, 27 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%); 4/72 objects misplaced (5.556%) 2024-04-16T11:46:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:51 smithi044 ceph-mon[93020]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:46:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:51 smithi044 ceph-mon[93020]: osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871] boot 2024-04-16T11:46:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:51 smithi044 ceph-mon[93020]: osdmap e55: 6 total, 6 up, 6 in 2024-04-16T11:46:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:51 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:46:51.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:51 smithi044 ceph-mon[93020]: osdmap e56: 6 total, 6 up, 6 in 2024-04-16T11:46:52.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:51 smithi045 ceph-mon[81423]: pgmap v74: 65 pgs: 8 active+clean+remapped, 8 active+undersized+degraded, 22 active+undersized, 27 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 15/72 objects degraded (20.833%); 4/72 objects misplaced (5.556%) 2024-04-16T11:46:52.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:51 smithi045 ceph-mon[81423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:46:52.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:51 smithi045 ceph-mon[81423]: osd.0 [v2:172.21.15.44:6802/123479871,v1:172.21.15.44:6803/123479871] boot 2024-04-16T11:46:52.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:51 smithi045 ceph-mon[81423]: osdmap e55: 6 total, 6 up, 6 in 2024-04-16T11:46:52.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:51 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T11:46:52.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:51 smithi045 ceph-mon[81423]: osdmap e56: 6 total, 6 up, 6 in 2024-04-16T11:46:53.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:52 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:53.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:52 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:53.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:52 smithi045 ceph-mon[81423]: osdmap e57: 6 total, 6 up, 6 in 2024-04-16T11:46:53.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:52 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:53.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:52 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:53.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:52 smithi044 ceph-mon[93020]: osdmap e57: 6 total, 6 up, 6 in 2024-04-16T11:46:53.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 -- 172.21.15.44:0/1685104819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c740ff000 msgr2=0x7f0c740ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:53.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 --2- 172.21.15.44:0/1685104819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c740ff000 0x7f0c740ff410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0c68004940 tx=0x7f0c68009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:53.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 -- 172.21.15.44:0/1685104819 shutdown_connections 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 --2- 172.21.15.44:0/1685104819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c741002b0 0x7f0c74100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 --2- 172.21.15.44:0/1685104819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c740ff000 0x7f0c740ff410 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 -- 172.21.15.44:0/1685104819 >> 172.21.15.44:0/1685104819 conn(0x7f0c740fa950 msgr2=0x7f0c740fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 -- 172.21.15.44:0/1685104819 shutdown_connections 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.720+0000 7f0c7c000700 1 -- 172.21.15.44:0/1685104819 wait complete. 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.721+0000 7f0c7c000700 1 Processor -- start 2024-04-16T11:46:53.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.721+0000 7f0c7c000700 1 -- start start 2024-04-16T11:46:53.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c7c000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740ff000 0x7f0c7410fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:53.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c7c000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 0x7f0c74110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:53.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c7c000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74110650 con 0x7f0c741002b0 2024-04-16T11:46:53.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c7c000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741107c0 con 0x7f0c740ff000 2024-04-16T11:46:53.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c78c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 0x7f0c74110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:53.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c78c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 0x7f0c74110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34280/0 (socket says 172.21.15.44:34280) 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c78c00700 1 -- 172.21.15.44:0/2537654595 learned_addr learned my addr 172.21.15.44:0/2537654595 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.722+0000 7f0c79600700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740ff000 0x7f0c7410fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c78c00700 1 -- 172.21.15.44:0/2537654595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740ff000 msgr2=0x7f0c7410fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c78c00700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740ff000 0x7f0c7410fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c78c00700 1 -- 172.21.15.44:0/2537654595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68005040 con 0x7f0c741002b0 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c78c00700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 0x7f0c74110050 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000c8c0 tx=0x7f0c600044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c6000a980 con 0x7f0c741002b0 2024-04-16T11:46:53.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c6000ce60 con 0x7f0c741002b0 2024-04-16T11:46:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.723+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c74115220 con 0x7f0c741002b0 2024-04-16T11:46:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.724+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c600115e0 con 0x7f0c741002b0 2024-04-16T11:46:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.724+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741157e0 con 0x7f0c741002b0 2024-04-16T11:46:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.725+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7404c710 con 0x7f0c741002b0 2024-04-16T11:46:53.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.726+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0c60012400 con 0x7f0c741002b0 2024-04-16T11:46:53.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.727+0000 7f0c71600700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0c5c06ec00 0x7f0c5c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:53.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.728+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f0c6008eac0 con 0x7f0c741002b0 2024-04-16T11:46:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.728+0000 7f0c79600700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0c5c06ec00 0x7f0c5c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:53.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.729+0000 7f0c79600700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0c5c06ec00 0x7f0c5c0710b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800b3b0 tx=0x7f0c68009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:53.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.733+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0c6005b2e0 con 0x7f0c741002b0 2024-04-16T11:46:53.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:53 smithi044 ceph-mon[93020]: pgmap v77: 65 pgs: 8 remapped+peering, 21 peering, 2 active+undersized+degraded, 7 active+undersized, 27 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T11:46:53.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:53 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T11:46:53.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.996+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c74104590 con 0x7f0c5c06ec00 2024-04-16T11:46:53.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:53.998+0000 7f0c71600700 1 -- 172.21.15.44:0/2537654595 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f0c74104590 con 0x7f0c5c06ec00 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.000+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0c5c06ec00 msgr2=0x7f0c5c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.000+0000 7f0c7c000700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0c5c06ec00 0x7f0c5c0710b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800b3b0 tx=0x7f0c68009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.000+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 msgr2=0x7f0c74110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.000+0000 7f0c7c000700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 0x7f0c74110050 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000c8c0 tx=0x7f0c600044c0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 shutdown_connections 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0c5c06ec00 0x7f0c5c0710b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c740ff000 0x7f0c7410fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 --2- 172.21.15.44:0/2537654595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c741002b0 0x7f0c74110050 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 >> 172.21.15.44:0/2537654595 conn(0x7f0c740fa950 msgr2=0x7f0c740fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 shutdown_connections 2024-04-16T11:46:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.001+0000 7f0c7c000700 1 -- 172.21.15.44:0/2537654595 wait complete. 2024-04-16T11:46:54.015 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:46:54.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:53 smithi045 ceph-mon[81423]: pgmap v77: 65 pgs: 8 remapped+peering, 21 peering, 2 active+undersized+degraded, 7 active+undersized, 27 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T11:46:54.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:53 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T11:46:54.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 -- 172.21.15.44:0/2562302781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 msgr2=0x7fe9740fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 --2- 172.21.15.44:0/2562302781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe9740fe460 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe964004940 tx=0x7fe964009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 -- 172.21.15.44:0/2562302781 shutdown_connections 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 --2- 172.21.15.44:0/2562302781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe9740fe460 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 --2- 172.21.15.44:0/2562302781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9740fce80 0x7fe9740fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 -- 172.21.15.44:0/2562302781 >> 172.21.15.44:0/2562302781 conn(0x7fe9740f8580 msgr2=0x7fe9740fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 -- 172.21.15.44:0/2562302781 shutdown_connections 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.162+0000 7fe97c800700 1 -- 172.21.15.44:0/2562302781 wait complete. 2024-04-16T11:46:54.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.163+0000 7fe97c800700 1 Processor -- start 2024-04-16T11:46:54.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.163+0000 7fe97c800700 1 -- start start 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe97c800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9740fce80 0x7fe97410fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe97c800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe974110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe97c800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe974110680 con 0x7fe9740fdff0 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe97c800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9741107f0 con 0x7fe9740fce80 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe974110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe974110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34296/0 (socket says 172.21.15.44:34296) 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979400700 1 -- 172.21.15.44:0/3908978915 learned_addr learned my addr 172.21.15.44:0/3908978915 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:54.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979e00700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9740fce80 0x7fe97410fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:54.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979400700 1 -- 172.21.15.44:0/3908978915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9740fce80 msgr2=0x7fe97410fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979400700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9740fce80 0x7fe97410fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.164+0000 7fe979400700 1 -- 172.21.15.44:0/3908978915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe964005040 con 0x7fe9740fdff0 2024-04-16T11:46:54.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.165+0000 7fe979400700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe974110080 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe96400b3b0 tx=0x7fe964009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:54.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.165+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe96400e040 con 0x7fe9740fdff0 2024-04-16T11:46:54.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.165+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe964004290 con 0x7fe9740fdff0 2024-04-16T11:46:54.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.165+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe96401c630 con 0x7fe9740fdff0 2024-04-16T11:46:54.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.165+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9741151f0 con 0x7fe9740fdff0 2024-04-16T11:46:54.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.165+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9741157b0 con 0x7fe9740fdff0 2024-04-16T11:46:54.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.167+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe974101980 con 0x7fe9740fdff0 2024-04-16T11:46:54.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.168+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe96400bbf0 con 0x7fe9740fdff0 2024-04-16T11:46:54.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.169+0000 7fe96e000700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe95c06ec00 0x7fe95c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:54.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.169+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7fe96408fd30 con 0x7fe9740fdff0 2024-04-16T11:46:54.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.169+0000 7fe979e00700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe95c06ec00 0x7fe95c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:54.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.170+0000 7fe979e00700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe95c06ec00 0x7fe95c0710b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe968004990 tx=0x7fe968009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:54.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.174+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe96405c5d0 con 0x7fe9740fdff0 2024-04-16T11:46:54.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.427+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe97405e400 con 0x7fe95c06ec00 2024-04-16T11:46:54.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.429+0000 7fe96e000700 1 -- 172.21.15.44:0/3908978915 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe97405e400 con 0x7fe95c06ec00 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe95c06ec00 msgr2=0x7fe95c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe95c06ec00 0x7fe95c0710b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe968004990 tx=0x7fe968009ee0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 msgr2=0x7fe974110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe974110080 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe96400b3b0 tx=0x7fe964009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 shutdown_connections 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe95c06ec00 0x7fe95c0710b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9740fce80 0x7fe97410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 --2- 172.21.15.44:0/3908978915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9740fdff0 0x7fe974110080 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.432+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 >> 172.21.15.44:0/3908978915 conn(0x7fe9740f8580 msgr2=0x7fe9740f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:54.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.433+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 shutdown_connections 2024-04-16T11:46:54.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.433+0000 7fe97c800700 1 -- 172.21.15.44:0/3908978915 wait complete. 2024-04-16T11:46:54.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 -- 172.21.15.44:0/941618141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80662f0 msgr2=0x7fa2a80ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 --2- 172.21.15.44:0/941618141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80662f0 0x7fa2a80ffa30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa298004940 tx=0x7fa298009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 -- 172.21.15.44:0/941618141 shutdown_connections 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 --2- 172.21.15.44:0/941618141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2a80fff70 0x7fa2a8102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 --2- 172.21.15.44:0/941618141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80662f0 0x7fa2a80ffa30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 -- 172.21.15.44:0/941618141 >> 172.21.15.44:0/941618141 conn(0x7fa2a80f74c0 msgr2=0x7fa2a80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.588+0000 7fa2ae000700 1 -- 172.21.15.44:0/941618141 shutdown_connections 2024-04-16T11:46:54.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.589+0000 7fa2ae000700 1 -- 172.21.15.44:0/941618141 wait complete. 2024-04-16T11:46:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.589+0000 7fa2ae000700 1 Processor -- start 2024-04-16T11:46:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2ae000700 1 -- start start 2024-04-16T11:46:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2ae000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2a80662f0 0x7fa2a80fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:54.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2ae000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 0x7fa2a80fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2ae000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80fbee0 con 0x7fa2a80fff70 2024-04-16T11:46:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2ae000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80fc050 con 0x7fa2a80662f0 2024-04-16T11:46:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2a6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 0x7fa2a80fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2a6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 0x7fa2a80fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34324/0 (socket says 172.21.15.44:34324) 2024-04-16T11:46:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.590+0000 7fa2a6a00700 1 -- 172.21.15.44:0/604218790 learned_addr learned my addr 172.21.15.44:0/604218790 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:54.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.591+0000 7fa2a7400700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2a80662f0 0x7fa2a80fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.591+0000 7fa2a6a00700 1 -- 172.21.15.44:0/604218790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2a80662f0 msgr2=0x7fa2a80fd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.591+0000 7fa2a6a00700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2a80662f0 0x7fa2a80fd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.591+0000 7fa2a6a00700 1 -- 172.21.15.44:0/604218790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa298005040 con 0x7fa2a80fff70 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.591+0000 7fa2a6a00700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 0x7fa2a80fb930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa294002a50 tx=0x7fa2940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.592+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa294007cc0 con 0x7fa2a80fff70 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.592+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29400f040 con 0x7fa2a80fff70 2024-04-16T11:46:54.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.592+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a80fc330 con 0x7fa2a80fff70 2024-04-16T11:46:54.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.592+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa294009640 con 0x7fa2a80fff70 2024-04-16T11:46:54.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.592+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a8115a90 con 0x7fa2a80fff70 2024-04-16T11:46:54.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.593+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a805f5f0 con 0x7fa2a80fff70 2024-04-16T11:46:54.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.594+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa2940097c0 con 0x7fa2a80fff70 2024-04-16T11:46:54.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.595+0000 7fa29f400700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa28c06ec10 0x7fa28c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:54.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.595+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7fa29408e6c0 con 0x7fa2a80fff70 2024-04-16T11:46:54.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.596+0000 7fa2a7400700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa28c06ec10 0x7fa28c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:54.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.597+0000 7fa2a7400700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa28c06ec10 0x7fa28c0710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa298000c00 tx=0x7fa298009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:54.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.604+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa2940573d0 con 0x7fa2a80fff70 2024-04-16T11:46:54.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.879+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2a80f8fb0 con 0x7fa28c06ec10 2024-04-16T11:46:54.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.902+0000 7fa29f400700 1 -- 172.21.15.44:0/604218790 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fa2a80f8fb0 con 0x7fa28c06ec10 2024-04-16T11:46:54.900 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:46:54.900 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (4m) 3s ago 14m 14.1M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:46:54.900 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (49s) 3s ago 14m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:46:54.900 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (42s) 37s ago 13m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:46:54.900 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (3m) 3s ago 14m 74.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (7m) 3s ago 7m 13.7M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (7m) 3s ago 7m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (7m) 37s ago 7m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (7m) 37s ago 7m 13.4M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (3m) 3s ago 16m 499M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (2m) 37s ago 13m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (2m) 3s ago 16m 43.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (110s) 37s ago 13m 31.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (4m) 3s ago 14m 7805k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (3m) 37s ago 13m 8200k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (10s) 3s ago 12m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (11m) 3s ago 11m 50.0M 1559M 17.2.7-847-g45317f95 b333581ccdf5 e8ad519010a9 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (10m) 3s ago 10m 46.7M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (10m) 37s ago 10m 46.4M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:46:54.901 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (9m) 37s ago 9m 44.8M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:46:54.902 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (9m) 37s ago 9m 46.6M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:46:54.902 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (2m) 3s ago 13m 55.0M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa28c06ec10 msgr2=0x7fa28c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa28c06ec10 0x7fa28c0710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa298000c00 tx=0x7fa298009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 msgr2=0x7fa2a80fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 0x7fa2a80fb930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa294002a50 tx=0x7fa2940046b0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 shutdown_connections 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa28c06ec10 0x7fa28c0710c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa2a80662f0 0x7fa2a80fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 --2- 172.21.15.44:0/604218790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2a80fff70 0x7fa2a80fb930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:54.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 >> 172.21.15.44:0/604218790 conn(0x7fa2a80f74c0 msgr2=0x7fa2a80f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:54.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 shutdown_connections 2024-04-16T11:46:54.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:54.905+0000 7fa2ae000700 1 -- 172.21.15.44:0/604218790 wait complete. 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.068+0000 7fe020c00700 1 -- 172.21.15.44:0/2312192531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 msgr2=0x7fe01c0ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.068+0000 7fe020c00700 1 --2- 172.21.15.44:0/2312192531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c0ff800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe00c004940 tx=0x7fe00c009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.069+0000 7fe020c00700 1 -- 172.21.15.44:0/2312192531 shutdown_connections 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.069+0000 7fe020c00700 1 --2- 172.21.15.44:0/2312192531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0ffd40 0x7fe01c102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.069+0000 7fe020c00700 1 --2- 172.21.15.44:0/2312192531 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c0ff800 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.069+0000 7fe020c00700 1 -- 172.21.15.44:0/2312192531 >> 172.21.15.44:0/2312192531 conn(0x7fe01c0f7270 msgr2=0x7fe01c0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:55.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.069+0000 7fe020c00700 1 -- 172.21.15.44:0/2312192531 shutdown_connections 2024-04-16T11:46:55.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.069+0000 7fe020c00700 1 -- 172.21.15.44:0/2312192531 wait complete. 2024-04-16T11:46:55.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.070+0000 7fe020c00700 1 Processor -- start 2024-04-16T11:46:55.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.071+0000 7fe020c00700 1 -- start start 2024-04-16T11:46:55.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.071+0000 7fe020c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c10fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:55.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.071+0000 7fe020c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0ffd40 0x7fe01c110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:55.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.071+0000 7fe020c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c110610 con 0x7fe01c0662f0 2024-04-16T11:46:55.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.071+0000 7fe020c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c110e60 con 0x7fe01c0ffd40 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.072+0000 7fe01b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c10fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.072+0000 7fe01b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c10fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34336/0 (socket says 172.21.15.44:34336) 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.072+0000 7fe01b400700 1 -- 172.21.15.44:0/2210028088 learned_addr learned my addr 172.21.15.44:0/2210028088 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.072+0000 7fe01b400700 1 -- 172.21.15.44:0/2210028088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0ffd40 msgr2=0x7fe01c110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.072+0000 7fe01b400700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0ffd40 0x7fe01c110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.072+0000 7fe01b400700 1 -- 172.21.15.44:0/2210028088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c005040 con 0x7fe01c0662f0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.073+0000 7fe01b400700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c10fad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00b970 tx=0x7fe00c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.073+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00c00e040 con 0x7fe01c0662f0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.073+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe00c004290 con 0x7fe01c0662f0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.073+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00c01c630 con 0x7fe01c0662f0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.073+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c1110e0 con 0x7fe01c0662f0 2024-04-16T11:46:55.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.073+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c111620 con 0x7fe01c0662f0 2024-04-16T11:46:55.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.074+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c05f5f0 con 0x7fe01c0662f0 2024-04-16T11:46:55.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.076+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe00c00bbf0 con 0x7fe01c0662f0 2024-04-16T11:46:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.076+0000 7fe013400700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe00006ec10 0x7fe0000710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:55.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.077+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7fe00c08fc60 con 0x7fe01c0662f0 2024-04-16T11:46:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.077+0000 7fe01aa00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe00006ec10 0x7fe0000710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:55.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.078+0000 7fe01aa00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe00006ec10 0x7fe0000710c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe008004d00 tx=0x7fe008009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:55.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.083+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe00c05c500 con 0x7fe01c0662f0 2024-04-16T11:46:55.175 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:55.175 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:55.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:55.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:55.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.420+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe01c05f5f0 con 0x7fe01c0662f0 2024-04-16T11:46:55.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.422+0000 7fe013400700 1 -- 172.21.15.44:0/2210028088 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fe00c013e60 con 0x7fe01c0662f0 2024-04-16T11:46:55.420 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:46:55.420 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:46:55.420 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:46:55.420 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 5, 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 9, 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 5 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:46:55.421 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:46:55.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe00006ec10 msgr2=0x7fe0000710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe00006ec10 0x7fe0000710c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe008004d00 tx=0x7fe008009ce0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 msgr2=0x7fe01c10fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c10fad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00b970 tx=0x7fe00c009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:55.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 shutdown_connections 2024-04-16T11:46:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe00006ec10 0x7fe0000710c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe01c0662f0 0x7fe01c10fad0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 --2- 172.21.15.44:0/2210028088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe01c0ffd40 0x7fe01c110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 >> 172.21.15.44:0/2210028088 conn(0x7fe01c0f7270 msgr2=0x7fe01c0fb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 shutdown_connections 2024-04-16T11:46:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.425+0000 7fe020c00700 1 -- 172.21.15.44:0/2210028088 wait complete. 2024-04-16T11:46:55.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.584+0000 7f42ad800700 1 -- 172.21.15.44:0/4094322328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a8103400 msgr2=0x7f42a8103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.584+0000 7f42ad800700 1 --2- 172.21.15.44:0/4094322328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a8103400 0x7f42a8103810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4298004940 tx=0x7f4298009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.584+0000 7f42ad800700 1 -- 172.21.15.44:0/4094322328 shutdown_connections 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.584+0000 7f42ad800700 1 --2- 172.21.15.44:0/4094322328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a81046b0 0x7f42a8104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.584+0000 7f42ad800700 1 --2- 172.21.15.44:0/4094322328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a8103400 0x7f42a8103810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.584+0000 7f42ad800700 1 -- 172.21.15.44:0/4094322328 >> 172.21.15.44:0/4094322328 conn(0x7f42a80725e0 msgr2=0x7f42a8074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.585+0000 7f42ad800700 1 -- 172.21.15.44:0/4094322328 shutdown_connections 2024-04-16T11:46:55.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.585+0000 7f42ad800700 1 -- 172.21.15.44:0/4094322328 wait complete. 2024-04-16T11:46:55.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.586+0000 7f42ad800700 1 Processor -- start 2024-04-16T11:46:55.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.586+0000 7f42ad800700 1 -- start start 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.586+0000 7f42ad800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a8103400 0x7f42a8113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.586+0000 7f42ad800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 0x7f42a8114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.586+0000 7f42ad800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a8114a60 con 0x7f42a81046b0 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.586+0000 7f42ad800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a8114bd0 con 0x7f42a8103400 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 0x7f42a8114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 0x7f42a8114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34344/0 (socket says 172.21.15.44:34344) 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 -- 172.21.15.44:0/3761306127 learned_addr learned my addr 172.21.15.44:0/3761306127 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6a00700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a8103400 0x7f42a8113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:55.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 -- 172.21.15.44:0/3761306127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a8103400 msgr2=0x7f42a8113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a8103400 0x7f42a8113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 -- 172.21.15.44:0/3761306127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298005040 con 0x7f42a81046b0 2024-04-16T11:46:55.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6a00700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a8103400 0x7f42a8113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:46:55.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.587+0000 7f42a6000700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 0x7f42a8114460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f429400c830 tx=0x7f4294004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:55.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.588+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f429400a760 con 0x7f42a81046b0 2024-04-16T11:46:55.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.588+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a8119630 con 0x7f42a81046b0 2024-04-16T11:46:55.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.588+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a8119bf0 con 0x7f42a81046b0 2024-04-16T11:46:55.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.588+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f429400ad90 con 0x7f42a81046b0 2024-04-16T11:46:55.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.588+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42940113a0 con 0x7f42a81046b0 2024-04-16T11:46:55.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.589+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42880052e0 con 0x7f42a81046b0 2024-04-16T11:46:55.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.590+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f429400a8c0 con 0x7f42a81046b0 2024-04-16T11:46:55.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.591+0000 7f429ea00700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f428c06ec00 0x7f428c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:55.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.591+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f429408e8b0 con 0x7f42a81046b0 2024-04-16T11:46:55.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.592+0000 7f42a6a00700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f428c06ec00 0x7f428c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:55.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.592+0000 7f42a6a00700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f428c06ec00 0x7f428c0710b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f429800b7a0 tx=0x7f4298009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:55.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.597+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f429405b150 con 0x7f42a81046b0 2024-04-16T11:46:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.880+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4288005da0 con 0x7f42a81046b0 2024-04-16T11:46:55.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.881+0000 7f429ea00700 1 -- 172.21.15.44:0/3761306127 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f429405ace0 con 0x7f42a81046b0 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:46:55.880 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:46:55.881 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:46:55.882 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:55.883 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:55.883 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:46:55.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.883+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f428c06ec00 msgr2=0x7f428c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.883+0000 7f42ad800700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f428c06ec00 0x7f428c0710b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f429800b7a0 tx=0x7f4298009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:55.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.883+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 msgr2=0x7f42a8114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.883+0000 7f42ad800700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 0x7f42a8114460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f429400c830 tx=0x7f4294004470 comp rx=0 tx=0).stop 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 shutdown_connections 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f428c06ec00 0x7f428c0710b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42a8103400 0x7f42a8113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 --2- 172.21.15.44:0/3761306127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42a81046b0 0x7f42a8114460 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 >> 172.21.15.44:0/3761306127 conn(0x7f42a80725e0 msgr2=0x7f42a8073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 shutdown_connections 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:55.884+0000 7f42ad800700 1 -- 172.21.15.44:0/3761306127 wait complete. 2024-04-16T11:46:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: pgmap v79: 65 pgs: 5 remapped+peering, 21 peering, 39 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s, 0 objects/s recovering 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded) 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:55.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:55 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2210028088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.041+0000 7f8e7f600700 1 -- 172.21.15.44:0/3408280484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 msgr2=0x7f8e780ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.041+0000 7f8e7f600700 1 --2- 172.21.15.44:0/3408280484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e780ffa30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e68004940 tx=0x7f8e68009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.042+0000 7f8e7f600700 1 -- 172.21.15.44:0/3408280484 shutdown_connections 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.042+0000 7f8e7f600700 1 --2- 172.21.15.44:0/3408280484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e780fff70 0x7f8e78102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.042+0000 7f8e7f600700 1 --2- 172.21.15.44:0/3408280484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e780ffa30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.042+0000 7f8e7f600700 1 -- 172.21.15.44:0/3408280484 >> 172.21.15.44:0/3408280484 conn(0x7f8e780f74c0 msgr2=0x7f8e780f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.042+0000 7f8e7f600700 1 -- 172.21.15.44:0/3408280484 shutdown_connections 2024-04-16T11:46:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.042+0000 7f8e7f600700 1 -- 172.21.15.44:0/3408280484 wait complete. 2024-04-16T11:46:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7f600700 1 Processor -- start 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7f600700 1 -- start start 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7f600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e7810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7f600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e780fff70 0x7f8e78110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7f600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e78110790 con 0x7f8e780662f0 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7f600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e78110fe0 con 0x7f8e780fff70 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e7810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e7810fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34362/0 (socket says 172.21.15.44:34362) 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.043+0000 7f8e7cc00700 1 -- 172.21.15.44:0/2098194230 learned_addr learned my addr 172.21.15.44:0/2098194230 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e6fe00700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e780fff70 0x7f8e78110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e7cc00700 1 -- 172.21.15.44:0/2098194230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e780fff70 msgr2=0x7f8e78110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e7cc00700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e780fff70 0x7f8e78110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e7cc00700 1 -- 172.21.15.44:0/2098194230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e68005040 con 0x7f8e780662f0 2024-04-16T11:46:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e7cc00700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e7810fc50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8e68000c00 tx=0x7f8e68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e6800e040 con 0x7f8e780662f0 2024-04-16T11:46:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e6800bcc0 con 0x7f8e780662f0 2024-04-16T11:46:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.044+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e78111260 con 0x7f8e780662f0 2024-04-16T11:46:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.045+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e6801c630 con 0x7f8e780662f0 2024-04-16T11:46:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.045+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e781117a0 con 0x7f8e780662f0 2024-04-16T11:46:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.046+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e780f8ff0 con 0x7f8e780662f0 2024-04-16T11:46:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.047+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e68013980 con 0x7f8e780662f0 2024-04-16T11:46:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.048+0000 7f8e75600700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8e5c06ec00 0x7f8e5c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.048+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f8e6808fce0 con 0x7f8e780662f0 2024-04-16T11:46:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.048+0000 7f8e6fe00700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8e5c06ec00 0x7f8e5c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.049+0000 7f8e6fe00700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8e5c06ec00 0x7f8e5c0710b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e64007fd0 tx=0x7f8e64009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.053+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8e6805c5a0 con 0x7f8e780662f0 2024-04-16T11:46:56.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.300+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e781119c0 con 0x7f8e5c06ec00 2024-04-16T11:46:56.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.302+0000 7f8e75600700 1 -- 172.21.15.44:0/2098194230 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f8e781119c0 con 0x7f8e5c06ec00 2024-04-16T11:46:56.300 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:46:56.300 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:46:56.300 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:46:56.300 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "7/21 daemons upgraded", 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:46:56.301 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8e5c06ec00 msgr2=0x7f8e5c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8e5c06ec00 0x7f8e5c0710b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e64007fd0 tx=0x7f8e64009040 comp rx=0 tx=0).stop 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 msgr2=0x7f8e7810fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e7810fc50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8e68000c00 tx=0x7f8e68009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 shutdown_connections 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8e5c06ec00 0x7f8e5c0710b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e780662f0 0x7f8e7810fc50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 --2- 172.21.15.44:0/2098194230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e780fff70 0x7f8e78110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 >> 172.21.15.44:0/2098194230 conn(0x7f8e780f74c0 msgr2=0x7f8e780fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:56.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 shutdown_connections 2024-04-16T11:46:56.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.305+0000 7f8e7f600700 1 -- 172.21.15.44:0/2098194230 wait complete. 2024-04-16T11:46:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: pgmap v79: 65 pgs: 5 remapped+peering, 21 peering, 39 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s, 0 objects/s recovering 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded) 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:46:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:55 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2210028088' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:46:56.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 -- 172.21.15.44:0/2598864906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fded0 msgr2=0x7f75b40fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 --2- 172.21.15.44:0/2598864906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fded0 0x7f75b40fe340 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75a0004940 tx=0x7f75a0009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:56.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 -- 172.21.15.44:0/2598864906 shutdown_connections 2024-04-16T11:46:56.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 --2- 172.21.15.44:0/2598864906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fded0 0x7f75b40fe340 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 --2- 172.21.15.44:0/2598864906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b40fcc20 0x7f75b40fd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 -- 172.21.15.44:0/2598864906 >> 172.21.15.44:0/2598864906 conn(0x7f75b40f83a0 msgr2=0x7f75b40fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:56.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 -- 172.21.15.44:0/2598864906 shutdown_connections 2024-04-16T11:46:56.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.452+0000 7f75b9000700 1 -- 172.21.15.44:0/2598864906 wait complete. 2024-04-16T11:46:56.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.453+0000 7f75b9000700 1 Processor -- start 2024-04-16T11:46:56.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.453+0000 7f75b9000700 1 -- start start 2024-04-16T11:46:56.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.453+0000 7f75b9000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 0x7f75b410d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:56.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.453+0000 7f75b9000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b40fded0 0x7f75b410dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:56.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.453+0000 7f75b9000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b410e360 con 0x7f75b40fcc20 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.453+0000 7f75b9000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b410ebb0 con 0x7f75b40fded0 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 0x7f75b410d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 0x7f75b410d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34370/0 (socket says 172.21.15.44:34370) 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 -- 172.21.15.44:0/2851506530 learned_addr learned my addr 172.21.15.44:0/2851506530 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 -- 172.21.15.44:0/2851506530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b40fded0 msgr2=0x7f75b410dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b40fded0 0x7f75b410dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 -- 172.21.15.44:0/2851506530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0005040 con 0x7f75b40fcc20 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.454+0000 7f75b3400700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 0x7f75b410d820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f75a4002a50 tx=0x7f75a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.455+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75a4007cc0 con 0x7f75b40fcc20 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.455+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75a400f040 con 0x7f75b40fcc20 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.455+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b410ee90 con 0x7f75b40fcc20 2024-04-16T11:46:56.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.455+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75a4009640 con 0x7f75b40fcc20 2024-04-16T11:46:56.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.455+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b410f3d0 con 0x7f75b40fcc20 2024-04-16T11:46:56.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.456+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b40fc020 con 0x7f75b40fcc20 2024-04-16T11:46:56.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.457+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75a40097a0 con 0x7f75b40fcc20 2024-04-16T11:46:56.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.458+0000 7f75ab400700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f759806ec00 0x7f75980710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:46:56.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.458+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f75a408e7c0 con 0x7f75b40fcc20 2024-04-16T11:46:56.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.458+0000 7f75b2a00700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f759806ec00 0x7f75980710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:46:56.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.459+0000 7f75b2a00700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f759806ec00 0x7f75980710b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f75a0004f90 tx=0x7f75a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:46:56.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.464+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f75a405af60 con 0x7f75b40fcc20 2024-04-16T11:46:56.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.800+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f75b404bed0 con 0x7f75b40fcc20 2024-04-16T11:46:56.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.801+0000 7f75ab400700 1 -- 172.21.15.44:0/2851506530 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f75a405aaf0 con 0x7f75b40fcc20 2024-04-16T11:46:56.800 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.804+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f759806ec00 msgr2=0x7f75980710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.804+0000 7f75b9000700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f759806ec00 0x7f75980710b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f75a0004f90 tx=0x7f75a0009e30 comp rx=0 tx=0).stop 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.804+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 msgr2=0x7f75b410d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.804+0000 7f75b9000700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 0x7f75b410d820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f75a4002a50 tx=0x7f75a40046b0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 shutdown_connections 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f759806ec00 0x7f75980710b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75b40fcc20 0x7f75b410d820 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 --2- 172.21.15.44:0/2851506530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75b40fded0 0x7f75b410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:46:56.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 >> 172.21.15.44:0/2851506530 conn(0x7f75b40f83a0 msgr2=0x7f75b40f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:46:56.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 shutdown_connections 2024-04-16T11:46:56.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:46:56.805+0000 7f75b9000700 1 -- 172.21.15.44:0/2851506530 wait complete. 2024-04-16T11:46:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:56 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3761306127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:46:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:56 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2851506530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:46:57.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:56 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3761306127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:46:57.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:56 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2851506530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:46:57.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 182, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104141 2024-04-16T11:46:57.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 182, success, outsize: 120 2024-04-16T11:46:57.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 184, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104233 2024-04-16T11:46:57.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 184, success, outsize: 120 2024-04-16T11:46:57.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 186, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104328 2024-04-16T11:46:57.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 186, success, outsize: 120 2024-04-16T11:46:57.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 188, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104429 2024-04-16T11:46:57.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 188, success, outsize: 120 2024-04-16T11:46:57.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 190, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104527 2024-04-16T11:46:57.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 190, success, outsize: 120 2024-04-16T11:46:57.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104619 2024-04-16T11:46:57.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 192, success, outsize: 120 2024-04-16T11:46:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104714 2024-04-16T11:46:58.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 194, success, outsize: 120 2024-04-16T11:46:58.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104815 2024-04-16T11:46:58.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 196, success, outsize: 120 2024-04-16T11:46:58.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 198, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104913 2024-04-16T11:46:58.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 198, success, outsize: 120 2024-04-16T11:46:58.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 200, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105005 2024-04-16T11:46:58.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 200, success, outsize: 120 2024-04-16T11:46:58.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 202, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105100 2024-04-16T11:46:58.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 202, success, outsize: 120 2024-04-16T11:46:58.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 204, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105201 2024-04-16T11:46:58.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 204, success, outsize: 120 2024-04-16T11:46:58.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 206, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105299 2024-04-16T11:46:58.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 206, success, outsize: 120 2024-04-16T11:46:58.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 208, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105391 2024-04-16T11:46:58.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 208, success, outsize: 120 2024-04-16T11:46:58.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:57 smithi044 ceph-mon[93020]: pgmap v80: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s, 0 objects/s recovering 2024-04-16T11:46:58.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:57 smithi044 ceph-mon[93020]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:58.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 210, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105486 2024-04-16T11:46:58.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 210, success, outsize: 120 2024-04-16T11:46:58.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 212, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105587 2024-04-16T11:46:58.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 212, success, outsize: 120 2024-04-16T11:46:58.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 214, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105685 2024-04-16T11:46:58.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 214, success, outsize: 120 2024-04-16T11:46:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:57 smithi045 ceph-mon[81423]: pgmap v80: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s, 0 objects/s recovering 2024-04-16T11:46:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:57 smithi045 ceph-mon[81423]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:46:58.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 216, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105777 2024-04-16T11:46:58.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 216, success, outsize: 120 2024-04-16T11:46:58.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 218, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105872 2024-04-16T11:46:58.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 218, success, outsize: 120 2024-04-16T11:46:58.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 220, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105973 2024-04-16T11:46:58.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 220, success, outsize: 120 2024-04-16T11:46:58.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 222, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106071 2024-04-16T11:46:58.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 222, success, outsize: 120 2024-04-16T11:46:58.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 224, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106163 2024-04-16T11:46:58.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 224, success, outsize: 120 2024-04-16T11:46:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 226, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106258 2024-04-16T11:46:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 226, success, outsize: 120 2024-04-16T11:46:58.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 228, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106359 2024-04-16T11:46:58.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 228, success, outsize: 120 2024-04-16T11:46:58.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 230, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106457 2024-04-16T11:46:58.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 230, success, outsize: 120 2024-04-16T11:46:58.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 232, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106549 2024-04-16T11:46:58.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 232, success, outsize: 120 2024-04-16T11:46:58.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 234, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106644 2024-04-16T11:46:58.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 234, success, outsize: 120 2024-04-16T11:46:58.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 236, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106745 2024-04-16T11:46:58.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 236, success, outsize: 120 2024-04-16T11:47:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:59 smithi044 ceph-mon[93020]: pgmap v81: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 893 B/s, 0 objects/s recovering 2024-04-16T11:47:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:47:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:46:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:59 smithi045 ceph-mon[81423]: pgmap v81: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 893 B/s, 0 objects/s recovering 2024-04-16T11:47:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:47:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:46:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:01.189 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:01.190 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:01.190 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:01.190 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:01.190 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T11:47:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T11:47:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:01 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T11:47:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:01 smithi044 ceph-mon[93020]: Upgrade: osd.1 is safe to restart 2024-04-16T11:47:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:01 smithi044 ceph-mon[93020]: pgmap v82: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 788 B/s, 0 objects/s recovering 2024-04-16T11:47:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:01 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T11:47:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:01 smithi045 ceph-mon[81423]: Upgrade: osd.1 is safe to restart 2024-04-16T11:47:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:01 smithi045 ceph-mon[81423]: pgmap v82: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 788 B/s, 0 objects/s recovering 2024-04-16T11:47:03.544 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:03 smithi044 ceph-mon[93020]: pgmap v83: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 669 B/s, 0 objects/s recovering 2024-04-16T11:47:03.545 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:03 smithi044 ceph-mon[93020]: Upgrade: Updating osd.1 2024-04-16T11:47:03.545 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:03 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:03.545 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:03 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T11:47:03.545 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:03 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:03.545 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:03 smithi044 ceph-mon[93020]: Deploying daemon osd.1 on smithi044 2024-04-16T11:47:03.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:03 smithi045 ceph-mon[81423]: pgmap v83: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 669 B/s, 0 objects/s recovering 2024-04-16T11:47:03.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:03 smithi045 ceph-mon[81423]: Upgrade: Updating osd.1 2024-04-16T11:47:03.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:03 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:03.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:03 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T11:47:03.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:03 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:03.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:03 smithi045 ceph-mon[81423]: Deploying daemon osd.1 on smithi044 2024-04-16T11:47:05.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:05 smithi044 ceph-mon[93020]: pgmap v84: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 583 B/s, 0 objects/s recovering 2024-04-16T11:47:05.431 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:05.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:05 smithi045 ceph-mon[81423]: pgmap v84: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 583 B/s, 0 objects/s recovering 2024-04-16T11:47:05.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:07.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:07 smithi044 ceph-mon[93020]: pgmap v85: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:07.476 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:07 smithi044 systemd[1]: Stopping Ceph osd.1 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:47:07.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:07 smithi045 ceph-mon[81423]: pgmap v85: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:07.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:07 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[55449]: 2024-04-16T11:47:07.623+0000 7f8c53400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:47:07.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:07 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[55449]: 2024-04-16T11:47:07.623+0000 7f8c53400700 -1 osd.1 57 *** Got signal Terminated *** 2024-04-16T11:47:07.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:07 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[55449]: 2024-04-16T11:47:07.623+0000 7f8c53400700 -1 osd.1 57 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T11:47:08.660 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:08 smithi044 ceph-mon[93020]: osd.1 marked itself down and dead 2024-04-16T11:47:08.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:08 smithi045 ceph-mon[81423]: osd.1 marked itself down and dead 2024-04-16T11:47:09.330 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:09 smithi044 podman[107207]: 2024-04-16 11:47:09.005678894 +0000 UTC m=+1.523391133 container died e8ad519010a99aa6c0afdcf61d3a895a3c26ad240c125a1683848ed26fecbe9b (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T11:47:09.698 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:09 smithi044 ceph-mon[93020]: pgmap v86: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:09.698 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:09 smithi044 ceph-mon[93020]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:47:09.698 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:09 smithi044 ceph-mon[93020]: osdmap e58: 6 total, 5 up, 6 in 2024-04-16T11:47:09.699 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:09 smithi044 podman[107207]: 2024-04-16 11:47:09.474517305 +0000 UTC m=+1.992229585 container cleanup e8ad519010a99aa6c0afdcf61d3a895a3c26ad240c125a1683848ed26fecbe9b (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-16T11:47:09.699 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:09 smithi044 bash[107207]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1 2024-04-16T11:47:09.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:09 smithi045 ceph-mon[81423]: pgmap v86: 65 pgs: 65 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:09.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:09 smithi045 ceph-mon[81423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:47:09.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:09 smithi045 ceph-mon[81423]: osdmap e58: 6 total, 5 up, 6 in 2024-04-16T11:47:09.975 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:09 smithi044 podman[107226]: 2024-04-16 11:47:09.700247557 +0000 UTC m=+0.680090252 container remove e8ad519010a99aa6c0afdcf61d3a895a3c26ad240c125a1683848ed26fecbe9b (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T11:47:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:10 smithi044 ceph-mon[93020]: osdmap e59: 6 total, 5 up, 6 in 2024-04-16T11:47:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:10.726 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:10 smithi044 podman[107315]: 2024-04-16 11:47:10.473784747 +0000 UTC m=+0.034325262 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:47:10.726 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:10 smithi044 podman[107315]: 2024-04-16 11:47:10.626468934 +0000 UTC m=+0.187009487 container create ed32a0d2efec256d2c6a50cd6eedadd78088191a191b7c9c04529ef3fc6d78c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:47:10.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:10 smithi045 ceph-mon[81423]: osdmap e59: 6 total, 5 up, 6 in 2024-04-16T11:47:10.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:10.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:11.226 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:10 smithi044 podman[107315]: 2024-04-16 11:47:10.88917729 +0000 UTC m=+0.449717847 container init ed32a0d2efec256d2c6a50cd6eedadd78088191a191b7c9c04529ef3fc6d78c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=) 2024-04-16T11:47:11.226 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:10 smithi044 podman[107315]: 2024-04-16 11:47:10.901600612 +0000 UTC m=+0.462141148 container start ed32a0d2efec256d2c6a50cd6eedadd78088191a191b7c9c04529ef3fc6d78c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T11:47:11.226 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:11 smithi044 podman[107315]: 2024-04-16 11:47:11.003809234 +0000 UTC m=+0.564349768 container attach ed32a0d2efec256d2c6a50cd6eedadd78088191a191b7c9c04529ef3fc6d78c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-16T11:47:11.719 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:11 smithi044 ceph-mon[93020]: pgmap v89: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:11.720 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:11 smithi044 ceph-mon[93020]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-16T11:47:11.720 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:11 smithi044 podman[107315]: 2024-04-16 11:47:11.371369813 +0000 UTC m=+0.931910380 container died ed32a0d2efec256d2c6a50cd6eedadd78088191a191b7c9c04529ef3fc6d78c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T11:47:11.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:11 smithi045 ceph-mon[81423]: pgmap v89: 65 pgs: 5 peering, 9 stale+active+clean, 51 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:11.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:11 smithi045 ceph-mon[81423]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-16T11:47:12.476 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 podman[107341]: 2024-04-16 11:47:12.082684205 +0000 UTC m=+0.692066119 container remove ed32a0d2efec256d2c6a50cd6eedadd78088191a191b7c9c04529ef3fc6d78c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-16T11:47:12.476 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-16T11:47:12.476 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 systemd[1]: Stopped Ceph osd.1 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:47:12.476 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.1.service: Consumed 13.757s CPU time. 2024-04-16T11:47:12.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:12 smithi045 ceph-mon[81423]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:12.945 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 systemd[1]: Starting Ceph osd.1 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:47:12.946 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:12 smithi044 ceph-mon[93020]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:13.226 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 podman[107439]: 2024-04-16 11:47:12.854717545 +0000 UTC m=+0.037065752 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:47:13.226 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:12 smithi044 podman[107439]: 2024-04-16 11:47:12.96724936 +0000 UTC m=+0.149597589 container create 0719d4b0fcca1c403e5a34f61f536cf8488d293c0253c5d623750c0ebe0d4fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate, org.label-schema.build-date=20240409, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T11:47:13.483 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:13 smithi044 podman[107439]: 2024-04-16 11:47:13.262755887 +0000 UTC m=+0.445104159 container init 0719d4b0fcca1c403e5a34f61f536cf8488d293c0253c5d623750c0ebe0d4fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T11:47:13.483 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:13 smithi044 podman[107439]: 2024-04-16 11:47:13.276930459 +0000 UTC m=+0.459278744 container start 0719d4b0fcca1c403e5a34f61f536cf8488d293c0253c5d623750c0ebe0d4fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, ceph=True) 2024-04-16T11:47:13.483 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:13 smithi044 podman[107439]: 2024-04-16 11:47:13.361357342 +0000 UTC m=+0.543705574 container attach 0719d4b0fcca1c403e5a34f61f536cf8488d293c0253c5d623750c0ebe0d4fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:47:13.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:13 smithi045 ceph-mon[81423]: pgmap v90: 65 pgs: 5 active+undersized+degraded, 5 peering, 22 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:13.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:13 smithi044 ceph-mon[93020]: pgmap v90: 65 pgs: 5 active+undersized+degraded, 5 peering, 22 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:14.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 238, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107564 2024-04-16T11:47:14.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 238, success, outsize: 120 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 bash[107439]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T11:47:14.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T11:47:14.478 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T11:47:14.478 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T11:47:14.478 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate[107453]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T11:47:14.478 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:14 smithi044 podman[107439]: 2024-04-16 11:47:14.265578708 +0000 UTC m=+1.447926946 container died 0719d4b0fcca1c403e5a34f61f536cf8488d293c0253c5d623750c0ebe0d4fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T11:47:14.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:14 smithi045 ceph-mon[81423]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-16T11:47:14.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:14 smithi044 ceph-mon[93020]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-16T11:47:15.424 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 podman[107592]: 2024-04-16 11:47:15.067938709 +0000 UTC m=+0.779931943 container remove 0719d4b0fcca1c403e5a34f61f536cf8488d293c0253c5d623750c0ebe0d4fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-16T11:47:15.708 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:15 smithi044 ceph-mon[93020]: pgmap v91: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:15.708 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 podman[107657]: 2024-04-16 11:47:15.334636273 +0000 UTC m=+0.039692702 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:47:15.708 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 podman[107657]: 2024-04-16 11:47:15.449199101 +0000 UTC m=+0.154255533 container create 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-16T11:47:15.708 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 podman[107657]: 2024-04-16 11:47:15.710874148 +0000 UTC m=+0.415930640 container init 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-16T11:47:15.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:15 smithi045 ceph-mon[81423]: pgmap v91: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:15.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 podman[107657]: 2024-04-16 11:47:15.722798395 +0000 UTC m=+0.427854832 container start 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T11:47:15.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 ceph-osd[107676]: -- 172.21.15.44:0/3499177811 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc667a1680 con 0x55dc6676ac00 2024-04-16T11:47:15.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 bash[107657]: 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d 2024-04-16T11:47:15.976 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:15 smithi044 systemd[1]: Started Ceph osd.1 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:47:16.725 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:16 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[107672]: 2024-04-16T11:47:16.297+0000 7f69ac1037c0 -1 Falling back to public interface 2024-04-16T11:47:17.299 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:17.299 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:17.299 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:17.299 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:17 smithi044 ceph-mon[93020]: pgmap v92: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:17.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:17.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:17 smithi045 ceph-mon[81423]: pgmap v92: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:19.526 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:19 smithi044 ceph-mon[93020]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:19.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:19 smithi045 ceph-mon[81423]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:20.185 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:19 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[107672]: 2024-04-16T11:47:19.925+0000 7f69ac1037c0 -1 osd.1 57 log_to_monitors true 2024-04-16T11:47:20.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:20 smithi044 ceph-mon[93020]: from='osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T11:47:20.476 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 11:47:20 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[107672]: 2024-04-16T11:47:20.294+0000 7f699b400700 -1 osd.1 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:47:20.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:20 smithi045 ceph-mon[81423]: from='osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T11:47:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:21 smithi044 ceph-mon[93020]: pgmap v94: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:21 smithi044 ceph-mon[93020]: from='osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T11:47:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:21 smithi044 ceph-mon[93020]: osdmap e60: 6 total, 5 up, 6 in 2024-04-16T11:47:21.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:21 smithi044 ceph-mon[93020]: from='osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:47:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:21 smithi045 ceph-mon[81423]: pgmap v94: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:21 smithi045 ceph-mon[81423]: from='osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T11:47:21.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:21 smithi045 ceph-mon[81423]: osdmap e60: 6 total, 5 up, 6 in 2024-04-16T11:47:21.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:21 smithi045 ceph-mon[81423]: from='osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:47:22.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:22 smithi045 ceph-mon[81423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:47:22.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:22 smithi045 ceph-mon[81423]: osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595] boot 2024-04-16T11:47:22.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:22 smithi045 ceph-mon[81423]: osdmap e61: 6 total, 6 up, 6 in 2024-04-16T11:47:22.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:22 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:47:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:22 smithi044 ceph-mon[93020]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:47:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:22 smithi044 ceph-mon[93020]: osd.1 [v2:172.21.15.44:6810/544437595,v1:172.21.15.44:6811/544437595] boot 2024-04-16T11:47:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:22 smithi044 ceph-mon[93020]: osdmap e61: 6 total, 6 up, 6 in 2024-04-16T11:47:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:22 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T11:47:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:23 smithi045 ceph-mon[81423]: pgmap v97: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:23.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:23 smithi045 ceph-mon[81423]: osdmap e62: 6 total, 6 up, 6 in 2024-04-16T11:47:23.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:23 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:23.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:23 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:23.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:23 smithi044 ceph-mon[93020]: pgmap v97: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:23.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:23 smithi044 ceph-mon[93020]: osdmap e62: 6 total, 6 up, 6 in 2024-04-16T11:47:23.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:23 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:23.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:23 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:25 smithi044 ceph-mon[93020]: pgmap v99: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:25.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:25 smithi045 ceph-mon[81423]: pgmap v99: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:25.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:26.442 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:26 smithi044 ceph-mon[93020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded) 2024-04-16T11:47:26.442 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:26 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:47:26.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:26 smithi045 ceph-mon[81423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded) 2024-04-16T11:47:26.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:26 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:47:26.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 -- 172.21.15.44:0/3248069768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000fffb0 msgr2=0x7f1700102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 --2- 172.21.15.44:0/3248069768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000fffb0 0x7f1700102390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f16ec004940 tx=0x7f16ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 -- 172.21.15.44:0/3248069768 shutdown_connections 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 --2- 172.21.15.44:0/3248069768 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000fffb0 0x7f1700102390 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 --2- 172.21.15.44:0/3248069768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17000662b0 0x7f17000ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 -- 172.21.15.44:0/3248069768 >> 172.21.15.44:0/3248069768 conn(0x7f17000f74c0 msgr2=0x7f17000f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 -- 172.21.15.44:0/3248069768 shutdown_connections 2024-04-16T11:47:26.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.988+0000 7f1708000700 1 -- 172.21.15.44:0/3248069768 wait complete. 2024-04-16T11:47:26.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.989+0000 7f1708000700 1 Processor -- start 2024-04-16T11:47:26.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.989+0000 7f1708000700 1 -- start start 2024-04-16T11:47:26.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1708000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 0x7f170010fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:26.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1708000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17000fffb0 0x7f1700110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:26.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1708000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700110770 con 0x7f17000662b0 2024-04-16T11:47:26.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1708000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700110fc0 con 0x7f17000fffb0 2024-04-16T11:47:26.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1705600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 0x7f170010fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:26.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1705600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 0x7f170010fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42528/0 (socket says 172.21.15.44:42528) 2024-04-16T11:47:26.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1705600700 1 -- 172.21.15.44:0/584866261 learned_addr learned my addr 172.21.15.44:0/584866261 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:26.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.990+0000 7f1704c00700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17000fffb0 0x7f1700110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.991+0000 7f1705600700 1 -- 172.21.15.44:0/584866261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17000fffb0 msgr2=0x7f1700110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.991+0000 7f1705600700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17000fffb0 0x7f1700110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.991+0000 7f1705600700 1 -- 172.21.15.44:0/584866261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec005040 con 0x7f17000662b0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.991+0000 7f1705600700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 0x7f170010fc30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c8b0 tx=0x7f16f4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.992+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f40106f0 con 0x7f17000662b0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.992+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17001112a0 con 0x7f17000662b0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.992+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16f4010d20 con 0x7f17000662b0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.992+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f4012e90 con 0x7f17000662b0 2024-04-16T11:47:26.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.992+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17001117e0 con 0x7f17000662b0 2024-04-16T11:47:26.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.994+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170005f5f0 con 0x7f17000662b0 2024-04-16T11:47:26.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.995+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16f4010850 con 0x7f17000662b0 2024-04-16T11:47:26.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.996+0000 7f16fd600700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f16e806ec10 0x7f16e80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:26.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.996+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f16f408e7e0 con 0x7f17000662b0 2024-04-16T11:47:26.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.996+0000 7f1704c00700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f16e806ec10 0x7f16e80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:26.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:26.997+0000 7f1704c00700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f16e806ec10 0x7f16e80710c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00b3b0 tx=0x7f16ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:26.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.001+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f16f405b0a0 con 0x7f17000662b0 2024-04-16T11:47:27.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.256+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1700111b70 con 0x7f16e806ec10 2024-04-16T11:47:27.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.259+0000 7f16fd600700 1 -- 172.21.15.44:0/584866261 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1700111b70 con 0x7f16e806ec10 2024-04-16T11:47:27.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f16e806ec10 msgr2=0x7f16e80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f16e806ec10 0x7f16e80710c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00b3b0 tx=0x7f16ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:27.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 msgr2=0x7f170010fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 0x7f170010fc30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c8b0 tx=0x7f16f4004470 comp rx=0 tx=0).stop 2024-04-16T11:47:27.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 shutdown_connections 2024-04-16T11:47:27.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f16e806ec10 0x7f16e80710c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17000662b0 0x7f170010fc30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 --2- 172.21.15.44:0/584866261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17000fffb0 0x7f1700110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 >> 172.21.15.44:0/584866261 conn(0x7f17000f74c0 msgr2=0x7f17000fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:27.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 shutdown_connections 2024-04-16T11:47:27.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.262+0000 7f1708000700 1 -- 172.21.15.44:0/584866261 wait complete. 2024-04-16T11:47:27.277 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:47:27.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 -- 172.21.15.44:0/3086846095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fce40 msgr2=0x7f24140fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 --2- 172.21.15.44:0/3086846095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fce40 0x7f24140fd250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2408004940 tx=0x7f2408009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:27.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 -- 172.21.15.44:0/3086846095 shutdown_connections 2024-04-16T11:47:27.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 --2- 172.21.15.44:0/3086846095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24140fe040 0x7f24140fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 --2- 172.21.15.44:0/3086846095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fce40 0x7f24140fd250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 -- 172.21.15.44:0/3086846095 >> 172.21.15.44:0/3086846095 conn(0x7f24140f8530 msgr2=0x7f24140fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:27.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 -- 172.21.15.44:0/3086846095 shutdown_connections 2024-04-16T11:47:27.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.423+0000 7f241c000700 1 -- 172.21.15.44:0/3086846095 wait complete. 2024-04-16T11:47:27.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.425+0000 7f241c000700 1 Processor -- start 2024-04-16T11:47:27.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.425+0000 7f241c000700 1 -- start start 2024-04-16T11:47:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.425+0000 7f241c000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24140fce40 0x7f2414071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.425+0000 7f241c000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 0x7f2414070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.425+0000 7f241c000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24140708c0 con 0x7f24140fe040 2024-04-16T11:47:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.425+0000 7f241c000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414070a30 con 0x7f24140fce40 2024-04-16T11:47:27.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.426+0000 7f2418c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 0x7f2414070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:27.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.426+0000 7f2418c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 0x7f2414070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42554/0 (socket says 172.21.15.44:42554) 2024-04-16T11:47:27.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.426+0000 7f2418c00700 1 -- 172.21.15.44:0/336153904 learned_addr learned my addr 172.21.15.44:0/336153904 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:27.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.426+0000 7f2419600700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24140fce40 0x7f2414071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:27.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f2418c00700 1 -- 172.21.15.44:0/336153904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24140fce40 msgr2=0x7f2414071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f2418c00700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24140fce40 0x7f2414071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f2418c00700 1 -- 172.21.15.44:0/336153904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2408005040 con 0x7f24140fe040 2024-04-16T11:47:27.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f2418c00700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 0x7f2414070310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f240000eb40 tx=0x7f240000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:27.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2400007cc0 con 0x7f24140fe040 2024-04-16T11:47:27.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f240000f040 con 0x7f24140fe040 2024-04-16T11:47:27.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.427+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2414070cb0 con 0x7f24140fe040 2024-04-16T11:47:27.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.428+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2414115a00 con 0x7f24140fe040 2024-04-16T11:47:27.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.428+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f240000ca10 con 0x7f24140fe040 2024-04-16T11:47:27.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.429+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241405f5f0 con 0x7f24140fe040 2024-04-16T11:47:27.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.430+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f240000cb70 con 0x7f24140fe040 2024-04-16T11:47:27.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.431+0000 7f2411600700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23fc06ec00 0x7f23fc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.432+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f240008e410 con 0x7f24140fe040 2024-04-16T11:47:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.432+0000 7f2419600700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23fc06ec00 0x7f23fc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:27.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.433+0000 7f2419600700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23fc06ec00 0x7f23fc0710b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2408013740 tx=0x7f2408009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:27.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.437+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2400057250 con 0x7f24140fe040 2024-04-16T11:47:27.593 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:27 smithi044 ceph-mon[93020]: pgmap v100: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:27.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.700+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2414102360 con 0x7f23fc06ec00 2024-04-16T11:47:27.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.703+0000 7f2411600700 1 -- 172.21.15.44:0/336153904 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2414102360 con 0x7f23fc06ec00 2024-04-16T11:47:27.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23fc06ec00 msgr2=0x7f23fc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23fc06ec00 0x7f23fc0710b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2408013740 tx=0x7f2408009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:27.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 msgr2=0x7f2414070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 0x7f2414070310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f240000eb40 tx=0x7f240000b040 comp rx=0 tx=0).stop 2024-04-16T11:47:27.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 shutdown_connections 2024-04-16T11:47:27.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23fc06ec00 0x7f23fc0710b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24140fce40 0x7f2414071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 --2- 172.21.15.44:0/336153904 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24140fe040 0x7f2414070310 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 >> 172.21.15.44:0/336153904 conn(0x7f24140f8530 msgr2=0x7f24140f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:27.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 shutdown_connections 2024-04-16T11:47:27.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.706+0000 7f241c000700 1 -- 172.21.15.44:0/336153904 wait complete. 2024-04-16T11:47:27.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:27 smithi045 ceph-mon[81423]: pgmap v100: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:27.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.867+0000 7fa616a00700 1 -- 172.21.15.44:0/3310291625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 msgr2=0x7fa6100ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.867+0000 7fa616a00700 1 --2- 172.21.15.44:0/3310291625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100ff800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc004940 tx=0x7fa5fc009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 -- 172.21.15.44:0/3310291625 shutdown_connections 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 --2- 172.21.15.44:0/3310291625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6100ffd40 0x7fa610102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 --2- 172.21.15.44:0/3310291625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100ff800 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 -- 172.21.15.44:0/3310291625 >> 172.21.15.44:0/3310291625 conn(0x7fa6100f7270 msgr2=0x7fa6100f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 -- 172.21.15.44:0/3310291625 shutdown_connections 2024-04-16T11:47:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 -- 172.21.15.44:0/3310291625 wait complete. 2024-04-16T11:47:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.868+0000 7fa616a00700 1 Processor -- start 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa616a00700 1 -- start start 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa616a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa616a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6100ffd40 0x7fa6100fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa616a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100fbcb0 con 0x7fa6100662f0 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa616a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100fbe20 con 0x7fa6100ffd40 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa615600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa615600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42572/0 (socket says 172.21.15.44:42572) 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.869+0000 7fa615600700 1 -- 172.21.15.44:0/3437754419 learned_addr learned my addr 172.21.15.44:0/3437754419 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa614c00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6100ffd40 0x7fa6100fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa615600700 1 -- 172.21.15.44:0/3437754419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6100ffd40 msgr2=0x7fa6100fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa615600700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6100ffd40 0x7fa6100fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa615600700 1 -- 172.21.15.44:0/3437754419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc005040 con 0x7fa6100662f0 2024-04-16T11:47:27.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa615600700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100fd230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00b970 tx=0x7fa5fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5fc00e040 con 0x7fa6100662f0 2024-04-16T11:47:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5fc00bbf0 con 0x7fa6100662f0 2024-04-16T11:47:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6100fc0a0 con 0x7fa6100662f0 2024-04-16T11:47:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.870+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5fc01c630 con 0x7fa6100662f0 2024-04-16T11:47:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.871+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6101158d0 con 0x7fa6100662f0 2024-04-16T11:47:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.872+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61005f5f0 con 0x7fa6100662f0 2024-04-16T11:47:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.873+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa5fc00bd60 con 0x7fa6100662f0 2024-04-16T11:47:27.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.874+0000 7fa60d600700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f806ec10 0x7fa5f80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:27.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.874+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7fa5fc08fcd0 con 0x7fa6100662f0 2024-04-16T11:47:27.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.874+0000 7fa614c00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f806ec10 0x7fa5f80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:27.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.875+0000 7fa614c00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f806ec10 0x7fa5f80710c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa604004e00 tx=0x7fa60400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:27.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:27.879+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa5fc058a60 con 0x7fa6100662f0 2024-04-16T11:47:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.129+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6100f8db0 con 0x7fa5f806ec10 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.147+0000 7fa60d600700 1 -- 172.21.15.44:0/3437754419 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa6100f8db0 con 0x7fa5f806ec10 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (4m) 5s ago 14m 13.2M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (83s) 5s ago 15m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (75s) 70s ago 13m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (4m) 5s ago 14m 74.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (7m) 5s ago 7m 13.9M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (7m) 5s ago 7m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:47:28.144 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (7m) 70s ago 7m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (7m) 70s ago 7m 13.4M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (4m) 5s ago 16m 500M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (3m) 70s ago 13m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (2m) 5s ago 17m 44.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (2m) 70s ago 13m 31.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (4m) 5s ago 15m 7717k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (4m) 70s ago 13m 8200k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (44s) 5s ago 12m 49.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (12s) 5s ago 12m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (11m) 5s ago 11m 48.5M 1559M 17.2.7-847-g45317f95 b333581ccdf5 ee8c49b5d5d1 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (10m) 70s ago 10m 46.4M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (10m) 70s ago 10m 44.8M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (9m) 70s ago 9m 46.6M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:47:28.145 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (3m) 5s ago 14m 55.7M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:47:28.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f806ec10 msgr2=0x7fa5f80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f806ec10 0x7fa5f80710c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa604004e00 tx=0x7fa60400b040 comp rx=0 tx=0).stop 2024-04-16T11:47:28.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 msgr2=0x7fa6100fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100fd230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00b970 tx=0x7fa5fc009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 shutdown_connections 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f806ec10 0x7fa5f80710c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6100662f0 0x7fa6100fd230 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 --2- 172.21.15.44:0/3437754419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6100ffd40 0x7fa6100fb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 >> 172.21.15.44:0/3437754419 conn(0x7fa6100f7270 msgr2=0x7fa6100f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 shutdown_connections 2024-04-16T11:47:28.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.150+0000 7fa616a00700 1 -- 172.21.15.44:0/3437754419 wait complete. 2024-04-16T11:47:28.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 -- 172.21.15.44:0/2566540207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 msgr2=0x7f75900ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 --2- 172.21.15.44:0/2566540207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f75900ffa30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7580009ce0 tx=0x7f7580004d10 comp rx=0 tx=0).stop 2024-04-16T11:47:28.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 -- 172.21.15.44:0/2566540207 shutdown_connections 2024-04-16T11:47:28.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 --2- 172.21.15.44:0/2566540207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75900fff70 0x7f7590102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 --2- 172.21.15.44:0/2566540207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f75900ffa30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 -- 172.21.15.44:0/2566540207 >> 172.21.15.44:0/2566540207 conn(0x7f75900f74c0 msgr2=0x7f75900f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:28.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 -- 172.21.15.44:0/2566540207 shutdown_connections 2024-04-16T11:47:28.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.307+0000 7f7594a00700 1 -- 172.21.15.44:0/2566540207 wait complete. 2024-04-16T11:47:28.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.308+0000 7f7594a00700 1 Processor -- start 2024-04-16T11:47:28.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.309+0000 7f7594a00700 1 -- start start 2024-04-16T11:47:28.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.309+0000 7f7594a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f759010fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:28.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.309+0000 7f7594a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75900fff70 0x7f75901103d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:28.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.309+0000 7f7594a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590111140 con 0x7f75900662f0 2024-04-16T11:47:28.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.309+0000 7f7594a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75901112b0 con 0x7f75900fff70 2024-04-16T11:47:28.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.309+0000 7f758de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f759010fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:28.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f759010fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42586/0 (socket says 172.21.15.44:42586) 2024-04-16T11:47:28.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758de00700 1 -- 172.21.15.44:0/2279073453 learned_addr learned my addr 172.21.15.44:0/2279073453 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:28.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758d400700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75900fff70 0x7f75901103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:28.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758de00700 1 -- 172.21.15.44:0/2279073453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75900fff70 msgr2=0x7f75901103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758de00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75900fff70 0x7f75901103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758de00700 1 -- 172.21.15.44:0/2279073453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7580009940 con 0x7f75900662f0 2024-04-16T11:47:28.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.310+0000 7f758de00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f759010fe90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7580009910 tx=0x7f7580004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:28.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.311+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758000e040 con 0x7f75900662f0 2024-04-16T11:47:28.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.311+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7580008e20 con 0x7f75900662f0 2024-04-16T11:47:28.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.311+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758001c740 con 0x7f75900662f0 2024-04-16T11:47:28.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.311+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75901114d0 con 0x7f75900662f0 2024-04-16T11:47:28.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.311+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7590111a90 con 0x7f75900662f0 2024-04-16T11:47:28.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.313+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75900f8ff0 con 0x7f75900662f0 2024-04-16T11:47:28.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.313+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7580012b70 con 0x7f75900662f0 2024-04-16T11:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.314+0000 7f7586000700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f757406ec10 0x7f75740710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.314+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f7580090460 con 0x7f75900662f0 2024-04-16T11:47:28.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.315+0000 7f758d400700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f757406ec10 0x7f75740710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:28.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.316+0000 7f758d400700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f757406ec10 0x7f75740710c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f757c004990 tx=0x7f757c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:28.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.320+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7580059150 con 0x7f75900662f0 2024-04-16T11:47:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:28 smithi044 ceph-mon[93020]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:28 smithi044 ceph-mon[93020]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.677+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f759005f5f0 con 0x7f75900662f0 2024-04-16T11:47:28.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.678+0000 7f7586000700 1 -- 172.21.15.44:0/2279073453 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f7580004e10 con 0x7f75900662f0 2024-04-16T11:47:28.675 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:47:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:47:28.675 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4, 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 8, 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:47:28.676 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:47:28.677 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f757406ec10 msgr2=0x7f75740710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f757406ec10 0x7f75740710c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f757c004990 tx=0x7f757c009ee0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 msgr2=0x7f759010fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f759010fe90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7580009910 tx=0x7f7580004d10 comp rx=0 tx=0).stop 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 shutdown_connections 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f757406ec10 0x7f75740710c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75900662f0 0x7f759010fe90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 --2- 172.21.15.44:0/2279073453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75900fff70 0x7f75901103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 >> 172.21.15.44:0/2279073453 conn(0x7f75900f74c0 msgr2=0x7f75900fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 shutdown_connections 2024-04-16T11:47:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.681+0000 7f7594a00700 1 -- 172.21.15.44:0/2279073453 wait complete. 2024-04-16T11:47:28.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:28 smithi045 ceph-mon[81423]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:28.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:28 smithi045 ceph-mon[81423]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:28.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 -- 172.21.15.44:0/3534866537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa70102370 msgr2=0x7ffa70104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 --2- 172.21.15.44:0/3534866537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa70102370 0x7ffa70104750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c004940 tx=0x7ffa5c009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:28.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 -- 172.21.15.44:0/3534866537 shutdown_connections 2024-04-16T11:47:28.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 --2- 172.21.15.44:0/3534866537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa70102370 0x7ffa70104750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 --2- 172.21.15.44:0/3534866537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa700662f0 0x7ffa70101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 -- 172.21.15.44:0/3534866537 >> 172.21.15.44:0/3534866537 conn(0x7ffa700f74c0 msgr2=0x7ffa700f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:28.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.837+0000 7ffa74e00700 1 -- 172.21.15.44:0/3534866537 shutdown_connections 2024-04-16T11:47:28.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.838+0000 7ffa74e00700 1 -- 172.21.15.44:0/3534866537 wait complete. 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.838+0000 7ffa74e00700 1 Processor -- start 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.838+0000 7ffa74e00700 1 -- start start 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa74e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 0x7ffa7010fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa74e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70102370 0x7ffa70110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa74e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa70110850 con 0x7ffa700662f0 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa74e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa70110f90 con 0x7ffa70102370 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa6de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 0x7ffa7010fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:28.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa6de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 0x7ffa7010fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42610/0 (socket says 172.21.15.44:42610) 2024-04-16T11:47:28.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.839+0000 7ffa6de00700 1 -- 172.21.15.44:0/2732128580 learned_addr learned my addr 172.21.15.44:0/2732128580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:28.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa6d400700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70102370 0x7ffa70110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:28.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa6de00700 1 -- 172.21.15.44:0/2732128580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70102370 msgr2=0x7ffa70110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:28.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa6de00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70102370 0x7ffa70110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa6de00700 1 -- 172.21.15.44:0/2732128580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c005040 con 0x7ffa700662f0 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa6de00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 0x7ffa7010fd10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000c7e0 tx=0x7ffa60007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa60012810 con 0x7ffa700662f0 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa60012e40 con 0x7ffa700662f0 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa6000a550 con 0x7ffa700662f0 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.840+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa70111270 con 0x7ffa700662f0 2024-04-16T11:47:28.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.841+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa70111830 con 0x7ffa700662f0 2024-04-16T11:47:28.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.842+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7005f5f0 con 0x7ffa700662f0 2024-04-16T11:47:28.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.843+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa60011380 con 0x7ffa700662f0 2024-04-16T11:47:28.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.844+0000 7ffa66000700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ffa5406ec10 0x7ffa540710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:28.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.844+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7ffa6008ed50 con 0x7ffa700662f0 2024-04-16T11:47:28.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.844+0000 7ffa6d400700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ffa5406ec10 0x7ffa540710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:28.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.845+0000 7ffa6d400700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ffa5406ec10 0x7ffa540710c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00b970 tx=0x7ffa5c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:28.854+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ffa6005b570 con 0x7ffa700662f0 2024-04-16T11:47:28.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 240, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109722 2024-04-16T11:47:28.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 240, success, outsize: 120 2024-04-16T11:47:29.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 242, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109804 2024-04-16T11:47:29.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 242, success, outsize: 120 2024-04-16T11:47:29.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 244, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109889 2024-04-16T11:47:29.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 244, success, outsize: 120 2024-04-16T11:47:29.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 246, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109980 2024-04-16T11:47:29.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 246, success, outsize: 120 2024-04-16T11:47:29.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 248, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110068 2024-04-16T11:47:29.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 248, success, outsize: 120 2024-04-16T11:47:29.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 250, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110150 2024-04-16T11:47:29.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 250, success, outsize: 120 2024-04-16T11:47:29.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 252, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110235 2024-04-16T11:47:29.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 252, success, outsize: 120 2024-04-16T11:47:29.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 254, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110326 2024-04-16T11:47:29.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 254, success, outsize: 120 2024-04-16T11:47:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.139+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffa70111c00 con 0x7ffa700662f0 2024-04-16T11:47:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.139+0000 7ffa66000700 1 -- 172.21.15.44:0/2732128580 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7ffa60011630 con 0x7ffa700662f0 2024-04-16T11:47:29.136 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:47:29.137 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:47:29.138 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:47:29.139 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:47:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ffa5406ec10 msgr2=0x7ffa540710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ffa5406ec10 0x7ffa540710c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00b970 tx=0x7ffa5c009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 msgr2=0x7ffa7010fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 0x7ffa7010fd10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000c7e0 tx=0x7ffa60007f90 comp rx=0 tx=0).stop 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 shutdown_connections 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ffa5406ec10 0x7ffa540710c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa700662f0 0x7ffa7010fd10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 --2- 172.21.15.44:0/2732128580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa70102370 0x7ffa70110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 >> 172.21.15.44:0/2732128580 conn(0x7ffa700f74c0 msgr2=0x7ffa700f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 shutdown_connections 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.142+0000 7ffa74e00700 1 -- 172.21.15.44:0/2732128580 wait complete. 2024-04-16T11:47:29.141 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:47:29.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 256, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110416 2024-04-16T11:47:29.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 256, success, outsize: 120 2024-04-16T11:47:29.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 258, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110499 2024-04-16T11:47:29.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 258, success, outsize: 120 2024-04-16T11:47:29.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 260, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110584 2024-04-16T11:47:29.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 260, success, outsize: 120 2024-04-16T11:47:29.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 262, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110675 2024-04-16T11:47:29.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 262, success, outsize: 120 2024-04-16T11:47:29.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 264, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110763 2024-04-16T11:47:29.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 264, success, outsize: 120 2024-04-16T11:47:29.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 266, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110845 2024-04-16T11:47:29.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 266, success, outsize: 120 2024-04-16T11:47:29.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 268, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110930 2024-04-16T11:47:29.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 268, success, outsize: 120 2024-04-16T11:47:29.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 270, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111021 2024-04-16T11:47:29.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 270, success, outsize: 120 2024-04-16T11:47:29.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 272, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111110 2024-04-16T11:47:29.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 272, success, outsize: 120 2024-04-16T11:47:29.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 274, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111194 2024-04-16T11:47:29.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 274, success, outsize: 120 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 -- 172.21.15.44:0/1867204709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0fff70 msgr2=0x7f48bc102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 --2- 172.21.15.44:0/1867204709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0fff70 0x7f48bc102390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f48a8004940 tx=0x7f48a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 -- 172.21.15.44:0/1867204709 shutdown_connections 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 --2- 172.21.15.44:0/1867204709 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0fff70 0x7f48bc102390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 --2- 172.21.15.44:0/1867204709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48bc0662f0 0x7f48bc0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 -- 172.21.15.44:0/1867204709 >> 172.21.15.44:0/1867204709 conn(0x7f48bc0f74c0 msgr2=0x7f48bc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 -- 172.21.15.44:0/1867204709 shutdown_connections 2024-04-16T11:47:29.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.294+0000 7f48c1c00700 1 -- 172.21.15.44:0/1867204709 wait complete. 2024-04-16T11:47:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.295+0000 7f48c1c00700 1 Processor -- start 2024-04-16T11:47:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.295+0000 7f48c1c00700 1 -- start start 2024-04-16T11:47:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48c1c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 0x7f48bc10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48c1c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48bc0fff70 0x7f48bc110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48c1c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc110850 con 0x7f48bc0662f0 2024-04-16T11:47:29.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48c1c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc110f90 con 0x7f48bc0fff70 2024-04-16T11:47:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48baa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 0x7f48bc10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48baa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 0x7f48bc10fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42634/0 (socket says 172.21.15.44:42634) 2024-04-16T11:47:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48baa00700 1 -- 172.21.15.44:0/3036431124 learned_addr learned my addr 172.21.15.44:0/3036431124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.296+0000 7f48ba000700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48bc0fff70 0x7f48bc110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:29.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48baa00700 1 -- 172.21.15.44:0/3036431124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48bc0fff70 msgr2=0x7f48bc110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48baa00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48bc0fff70 0x7f48bc110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48baa00700 1 -- 172.21.15.44:0/3036431124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a8005040 con 0x7f48bc0662f0 2024-04-16T11:47:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48baa00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 0x7f48bc10fd10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c800 tx=0x7f48ac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48ac00a760 con 0x7f48bc0662f0 2024-04-16T11:47:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48ac00ad90 con 0x7f48bc0662f0 2024-04-16T11:47:29.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc111270 con 0x7f48bc0662f0 2024-04-16T11:47:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48ac0113a0 con 0x7f48bc0662f0 2024-04-16T11:47:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.297+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc111830 con 0x7f48bc0662f0 2024-04-16T11:47:29.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.299+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc05f5f0 con 0x7f48bc0662f0 2024-04-16T11:47:29.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.301+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f48ac012400 con 0x7f48bc0662f0 2024-04-16T11:47:29.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.301+0000 7f48b2a00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f48a006ec10 0x7f48a00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:29.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.302+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f48ac08e930 con 0x7f48bc0662f0 2024-04-16T11:47:29.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.302+0000 7f48ba000700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f48a006ec10 0x7f48a00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:29.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.303+0000 7f48ba000700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f48a006ec10 0x7f48a00710c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48a8004f90 tx=0x7f48a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:29.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 276, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111296 2024-04-16T11:47:29.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.307+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f48ac05b1f0 con 0x7f48bc0662f0 2024-04-16T11:47:29.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 276, success, outsize: 120 2024-04-16T11:47:29.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 278, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111387 2024-04-16T11:47:29.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 278, success, outsize: 120 2024-04-16T11:47:29.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 280, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111475 2024-04-16T11:47:29.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 280, success, outsize: 120 2024-04-16T11:47:29.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 282, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111557 2024-04-16T11:47:29.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 282, success, outsize: 120 2024-04-16T11:47:29.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 284, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111642 2024-04-16T11:47:29.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 284, success, outsize: 120 2024-04-16T11:47:29.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 286, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111733 2024-04-16T11:47:29.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 286, success, outsize: 120 2024-04-16T11:47:29.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 288, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111821 2024-04-16T11:47:29.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 288, success, outsize: 120 2024-04-16T11:47:29.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 290, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111903 2024-04-16T11:47:29.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 290, success, outsize: 120 2024-04-16T11:47:29.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 292, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111988 2024-04-16T11:47:29.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 292, success, outsize: 120 2024-04-16T11:47:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 294, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112079 2024-04-16T11:47:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 294, success, outsize: 120 2024-04-16T11:47:29.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:29 smithi044 ceph-mon[93020]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:29.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:29 smithi044 ceph-mon[93020]: pgmap v101: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:29.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:29 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2279073453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:29.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:29 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2732128580' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:47:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.638+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f48bc111c10 con 0x7f48a006ec10 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.641+0000 7f48b2a00700 1 -- 172.21.15.44:0/3036431124 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f48bc111c10 con 0x7f48a006ec10 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:47:29.638 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:47:29.639 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:47:29.639 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:47:29.639 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "8/21 daemons upgraded", 2024-04-16T11:47:29.639 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T11:47:29.639 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:47:29.639 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:47:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f48a006ec10 msgr2=0x7f48a00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f48a006ec10 0x7f48a00710c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48a8004f90 tx=0x7f48a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 msgr2=0x7f48bc10fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 0x7f48bc10fd10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c800 tx=0x7f48ac004470 comp rx=0 tx=0).stop 2024-04-16T11:47:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 shutdown_connections 2024-04-16T11:47:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f48a006ec10 0x7f48a00710c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48bc0662f0 0x7f48bc10fd10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 --2- 172.21.15.44:0/3036431124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48bc0fff70 0x7f48bc110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 >> 172.21.15.44:0/3036431124 conn(0x7f48bc0f74c0 msgr2=0x7f48bc0fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 shutdown_connections 2024-04-16T11:47:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.644+0000 7f48c1c00700 1 -- 172.21.15.44:0/3036431124 wait complete. 2024-04-16T11:47:29.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 -- 172.21.15.44:0/3185398859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58740fe040 msgr2=0x7f58740fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 --2- 172.21.15.44:0/3185398859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58740fe040 0x7f58740fe4b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f585c004940 tx=0x7f585c009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:29.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 -- 172.21.15.44:0/3185398859 shutdown_connections 2024-04-16T11:47:29.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 --2- 172.21.15.44:0/3185398859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58740fe040 0x7f58740fe4b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 --2- 172.21.15.44:0/3185398859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 0x7f58740fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 -- 172.21.15.44:0/3185398859 >> 172.21.15.44:0/3185398859 conn(0x7f58740f8530 msgr2=0x7f58740fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:29.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 -- 172.21.15.44:0/3185398859 shutdown_connections 2024-04-16T11:47:29.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.805+0000 7f5879200700 1 -- 172.21.15.44:0/3185398859 wait complete. 2024-04-16T11:47:29.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.806+0000 7f5879200700 1 Processor -- start 2024-04-16T11:47:29.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.806+0000 7f5879200700 1 -- start start 2024-04-16T11:47:29.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.806+0000 7f5879200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 0x7f587410fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:29.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.806+0000 7f5879200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 0x7f5874111100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:29.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.806+0000 7f5879200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874110820 con 0x7f5874110360 2024-04-16T11:47:29.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.806+0000 7f5879200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874110960 con 0x7f58740fce40 2024-04-16T11:47:29.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5869400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 0x7f5874111100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5869400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 0x7f5874111100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42654/0 (socket says 172.21.15.44:42654) 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5869400700 1 -- 172.21.15.44:0/3475343658 learned_addr learned my addr 172.21.15.44:0/3475343658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5871e00700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 0x7f587410fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5869400700 1 -- 172.21.15.44:0/3475343658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 msgr2=0x7f587410fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5869400700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 0x7f587410fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.807+0000 7f5869400700 1 -- 172.21.15.44:0/3475343658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c005040 con 0x7f5874110360 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.808+0000 7f5871e00700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 0x7f587410fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:47:29.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.808+0000 7f5869400700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 0x7f5874111100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f585c00b970 tx=0x7f585c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.808+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585c00e040 con 0x7f5874110360 2024-04-16T11:47:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.808+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f585c004290 con 0x7f5874110360 2024-04-16T11:47:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.808+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5874111730 con 0x7f5874110360 2024-04-16T11:47:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.808+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585c01c3f0 con 0x7f5874110360 2024-04-16T11:47:29.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.809+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5874111c90 con 0x7f5874110360 2024-04-16T11:47:29.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.810+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58741060e0 con 0x7f5874110360 2024-04-16T11:47:29.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.811+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f585c00bbf0 con 0x7f5874110360 2024-04-16T11:47:29.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.813+0000 7f586aa00700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f585806ec10 0x7f58580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:47:29.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.813+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f585c08fcb0 con 0x7f5874110360 2024-04-16T11:47:29.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.813+0000 7f5871e00700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f585806ec10 0x7f58580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:47:29.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.814+0000 7f5871e00700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f585806ec10 0x7f58580710c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5864009fd0 tx=0x7f5864009f00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:47:29.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:29.818+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f585c0589a0 con 0x7f5874110360 2024-04-16T11:47:29.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:29 smithi045 ceph-mon[81423]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:29.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:29 smithi045 ceph-mon[81423]: pgmap v101: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:29.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:29 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2279073453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:29.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:29 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2732128580' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:47:30.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.149+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f587405f5f0 con 0x7f5874110360 2024-04-16T11:47:30.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.150+0000 7f586aa00700 1 -- 172.21.15.44:0/3475343658 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f585c05c060 con 0x7f5874110360 2024-04-16T11:47:30.147 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.153+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f585806ec10 msgr2=0x7f58580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.153+0000 7f5879200700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f585806ec10 0x7f58580710c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5864009fd0 tx=0x7f5864009f00 comp rx=0 tx=0).stop 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.153+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 msgr2=0x7f5874111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.153+0000 7f5879200700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 0x7f5874111100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f585c00b970 tx=0x7f585c009e30 comp rx=0 tx=0).stop 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 shutdown_connections 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f585806ec10 0x7f58580710c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58740fce40 0x7f587410fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 --2- 172.21.15.44:0/3475343658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5874110360 0x7f5874111100 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 >> 172.21.15.44:0/3475343658 conn(0x7f58740f8530 msgr2=0x7f58740f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:47:30.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 shutdown_connections 2024-04-16T11:47:30.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:47:30.154+0000 7f5879200700 1 -- 172.21.15.44:0/3475343658 wait complete. 2024-04-16T11:47:30.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:30 smithi044 ceph-mon[93020]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:30 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3475343658' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:47:30.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:30 smithi045 ceph-mon[81423]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:47:30.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:30 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3475343658' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: pgmap v102: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:31.961 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: pgmap v102: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:47:32.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T11:47:32.888 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:32 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T11:47:32.888 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:32 smithi044 ceph-mon[93020]: Upgrade: osd.2 is safe to restart 2024-04-16T11:47:33.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:32 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T11:47:33.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:32 smithi045 ceph-mon[81423]: Upgrade: osd.2 is safe to restart 2024-04-16T11:47:33.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:33 smithi044 ceph-mon[93020]: pgmap v103: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:33.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:33.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T11:47:33.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:33.981 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:33 smithi045 ceph-mon[81423]: pgmap v103: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:33.981 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:33.981 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T11:47:33.981 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:47:34.789 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:34 smithi044 ceph-mon[93020]: Upgrade: Updating osd.2 2024-04-16T11:47:34.789 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:34 smithi044 ceph-mon[93020]: Deploying daemon osd.2 on smithi044 2024-04-16T11:47:35.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:34 smithi045 ceph-mon[81423]: Upgrade: Updating osd.2 2024-04-16T11:47:35.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:34 smithi045 ceph-mon[81423]: Deploying daemon osd.2 on smithi044 2024-04-16T11:47:35.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:35 smithi044 ceph-mon[93020]: pgmap v104: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:35.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:35 smithi045 ceph-mon[81423]: pgmap v104: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:37.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:37 smithi044 ceph-mon[93020]: pgmap v105: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:37 smithi045 ceph-mon[81423]: pgmap v105: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:38.475 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:38 smithi044 systemd[1]: Stopping Ceph osd.2 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:47:38.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:38 smithi044 ceph-mon[93020]: osd.2 marked itself down and dead 2024-04-16T11:47:38.978 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:38 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[62930]: 2024-04-16T11:47:38.609+0000 7fafce000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:47:38.978 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:38 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[62930]: 2024-04-16T11:47:38.609+0000 7fafce000700 -1 osd.2 62 *** Got signal Terminated *** 2024-04-16T11:47:38.978 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:38 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[62930]: 2024-04-16T11:47:38.609+0000 7fafce000700 -1 osd.2 62 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T11:47:39.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:38 smithi045 ceph-mon[81423]: osd.2 marked itself down and dead 2024-04-16T11:47:39.476 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:39 smithi044 podman[112575]: 2024-04-16 11:47:39.19755109 +0000 UTC m=+0.718607768 container died ee8c49b5d5d16eaaf962320dffe022ce9b8cbf9f26ec794ee3cf6de3c6d6df14 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-16T11:47:39.901 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:39 smithi044 ceph-mon[93020]: pgmap v106: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:39.902 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:39 smithi044 ceph-mon[93020]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:47:39.902 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:39 smithi044 ceph-mon[93020]: osdmap e63: 6 total, 5 up, 6 in 2024-04-16T11:47:39.902 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:39 smithi044 podman[112575]: 2024-04-16 11:47:39.694725509 +0000 UTC m=+1.215782191 container cleanup ee8c49b5d5d16eaaf962320dffe022ce9b8cbf9f26ec794ee3cf6de3c6d6df14 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-16T11:47:39.902 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:39 smithi044 bash[112575]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2 2024-04-16T11:47:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:39 smithi045 ceph-mon[81423]: pgmap v106: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:39 smithi045 ceph-mon[81423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:47:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:39 smithi045 ceph-mon[81423]: osdmap e63: 6 total, 5 up, 6 in 2024-04-16T11:47:40.225 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:39 smithi044 podman[112596]: 2024-04-16 11:47:39.904162005 +0000 UTC m=+0.690366220 container remove ee8c49b5d5d16eaaf962320dffe022ce9b8cbf9f26ec794ee3cf6de3c6d6df14 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-04-16T11:47:40.959 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:40 smithi044 podman[112684]: 2024-04-16 11:47:40.658856689 +0000 UTC m=+0.039724671 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:47:40.959 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:40 smithi044 podman[112684]: 2024-04-16 11:47:40.80513077 +0000 UTC m=+0.185998779 container create b10f28e6afb0f697c00db58d5136f31b6dd9057a0523172a7eab509dc740742b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux ) 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: osdmap e64: 6 total, 5 up, 6 in 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [0, 5]}]: dispatch 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1", "id": [0, 5]}]: dispatch 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2, 0, 1]}]: dispatch 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [3, 2]}]: dispatch 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [0, 5]}]: dispatch 2024-04-16T11:47:40.960 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.14"}]: dispatch 2024-04-16T11:47:41.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: osdmap e64: 6 total, 5 up, 6 in 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [0, 5]}]: dispatch 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1", "id": [0, 5]}]: dispatch 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2, 0, 1]}]: dispatch 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [3, 2]}]: dispatch 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [0, 5]}]: dispatch 2024-04-16T11:47:41.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.14"}]: dispatch 2024-04-16T11:47:41.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:41 smithi044 podman[112684]: 2024-04-16 11:47:41.084543322 +0000 UTC m=+0.465411341 container init b10f28e6afb0f697c00db58d5136f31b6dd9057a0523172a7eab509dc740742b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-16T11:47:41.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:41 smithi044 podman[112684]: 2024-04-16 11:47:41.096966399 +0000 UTC m=+0.477834448 container start b10f28e6afb0f697c00db58d5136f31b6dd9057a0523172a7eab509dc740742b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T11:47:41.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:41 smithi044 podman[112684]: 2024-04-16 11:47:41.182811859 +0000 UTC m=+0.563679877 container attach b10f28e6afb0f697c00db58d5136f31b6dd9057a0523172a7eab509dc740742b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0) 2024-04-16T11:47:41.914 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:41 smithi044 podman[112684]: 2024-04-16 11:47:41.569385198 +0000 UTC m=+0.950253217 container died b10f28e6afb0f697c00db58d5136f31b6dd9057a0523172a7eab509dc740742b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: pgmap v109: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [0, 5]}]': finished 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1", "id": [0, 5]}]': finished 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2, 0, 1]}]': finished 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [3, 2]}]': finished 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [0, 5]}]': finished 2024-04-16T11:47:41.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.14"}]': finished 2024-04-16T11:47:41.915 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:41 smithi044 ceph-mon[93020]: osdmap e65: 6 total, 5 up, 6 in 2024-04-16T11:47:42.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: pgmap v109: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [0, 5]}]': finished 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1", "id": [0, 5]}]': finished 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [3, 2, 0, 1]}]': finished 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [3, 2]}]': finished 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [0, 5]}]': finished 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.14"}]': finished 2024-04-16T11:47:42.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:41 smithi045 ceph-mon[81423]: osdmap e65: 6 total, 5 up, 6 in 2024-04-16T11:47:42.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:42 smithi044 podman[112710]: 2024-04-16 11:47:42.328384468 +0000 UTC m=+0.739373230 container remove b10f28e6afb0f697c00db58d5136f31b6dd9057a0523172a7eab509dc740742b (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T11:47:42.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:42 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-16T11:47:42.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:42 smithi044 systemd[1]: Stopped Ceph osd.2 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:47:42.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:42 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2.service: Consumed 13.290s CPU time. 2024-04-16T11:47:43.144 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:42 smithi044 systemd[1]: Starting Ceph osd.2 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:47:43.464 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:43 smithi044 podman[112811]: 2024-04-16 11:47:43.053016879 +0000 UTC m=+0.036527387 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:47:43.464 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:43 smithi044 podman[112811]: 2024-04-16 11:47:43.17114423 +0000 UTC m=+0.154654770 container create 4cb953e53a1dbb553bcf4d74a128a3359d10a0aca0630b01682f2cd0fbb50790 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T11:47:43.465 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:43 smithi044 ceph-mon[93020]: pgmap v111: 65 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 18 active+undersized, 40 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T11:47:43.465 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:43 smithi044 ceph-mon[93020]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T11:47:43.465 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:43 smithi044 ceph-mon[93020]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 4 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:43 smithi045 ceph-mon[81423]: pgmap v111: 65 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 18 active+undersized, 40 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-04-16T11:47:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:43 smithi045 ceph-mon[81423]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T11:47:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:43 smithi045 ceph-mon[81423]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 4 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:43.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:43 smithi044 podman[112811]: 2024-04-16 11:47:43.467188689 +0000 UTC m=+0.450699235 container init 4cb953e53a1dbb553bcf4d74a128a3359d10a0aca0630b01682f2cd0fbb50790 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-16T11:47:43.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:43 smithi044 podman[112811]: 2024-04-16 11:47:43.480329376 +0000 UTC m=+0.463839897 container start 4cb953e53a1dbb553bcf4d74a128a3359d10a0aca0630b01682f2cd0fbb50790 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-16T11:47:43.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:43 smithi044 podman[112811]: 2024-04-16 11:47:43.557151903 +0000 UTC m=+0.540662421 container attach 4cb953e53a1dbb553bcf4d74a128a3359d10a0aca0630b01682f2cd0fbb50790 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T11:47:44.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 296, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112905 2024-04-16T11:47:44.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 296, success, outsize: 120 2024-04-16T11:47:44.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:44 smithi045 ceph-mon[81423]: osdmap e67: 6 total, 5 up, 6 in 2024-04-16T11:47:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-mon[93020]: osdmap e67: 6 total, 5 up, 6 in 2024-04-16T11:47:44.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T11:47:44.726 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 bash[112811]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T11:47:44.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T11:47:44.728 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T11:47:44.728 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T11:47:44.728 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate[112826]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T11:47:44.728 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:44 smithi044 podman[112811]: 2024-04-16 11:47:44.444794125 +0000 UTC m=+1.428304671 container died 4cb953e53a1dbb553bcf4d74a128a3359d10a0aca0630b01682f2cd0fbb50790 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-16T11:47:45.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:45 smithi044 ceph-mon[93020]: pgmap v114: 65 pgs: 6 active+undersized+degraded, 24 active+undersized, 35 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:45.475 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:45 smithi044 podman[112949]: 2024-04-16 11:47:45.227448638 +0000 UTC m=+0.761428651 container remove 4cb953e53a1dbb553bcf4d74a128a3359d10a0aca0630b01682f2cd0fbb50790 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-16T11:47:45.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:45 smithi045 ceph-mon[81423]: pgmap v114: 65 pgs: 6 active+undersized+degraded, 24 active+undersized, 35 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T11:47:45.963 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:45 smithi044 podman[113008]: 2024-04-16 11:47:45.489049089 +0000 UTC m=+0.036130660 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:47:45.964 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:45 smithi044 podman[113008]: 2024-04-16 11:47:45.619944879 +0000 UTC m=+0.167026475 container create b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T11:47:45.964 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:45 smithi044 podman[113008]: 2024-04-16 11:47:45.966201658 +0000 UTC m=+0.513283294 container init b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-16T11:47:46.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:45 smithi044 podman[113008]: 2024-04-16 11:47:45.978133524 +0000 UTC m=+0.525215099 container start b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0) 2024-04-16T11:47:46.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:46 smithi044 ceph-osd[113027]: -- 172.21.15.44:0/2525975362 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a4490f3680 con 0x55a4490bcc00 2024-04-16T11:47:46.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:46 smithi044 bash[113008]: b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 2024-04-16T11:47:46.226 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:46 smithi044 systemd[1]: Started Ceph osd.2 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:47:46.975 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:46 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[113023]: 2024-04-16T11:47:46.558+0000 7fef431877c0 -1 Falling back to public interface 2024-04-16T11:47:47.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:47 smithi044 ceph-mon[93020]: pgmap v115: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 1.1 KiB/s, 0 objects/s recovering 2024-04-16T11:47:47.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:47.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:47.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:47.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:47 smithi045 ceph-mon[81423]: pgmap v115: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 1.1 KiB/s, 0 objects/s recovering 2024-04-16T11:47:47.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:47.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:47.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:47:49.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:49 smithi044 ceph-mon[93020]: pgmap v116: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 962 B/s, 0 objects/s recovering 2024-04-16T11:47:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:49 smithi045 ceph-mon[81423]: pgmap v116: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 962 B/s, 0 objects/s recovering 2024-04-16T11:47:50.561 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:50 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:50.641 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:50 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[113023]: 2024-04-16T11:47:50.254+0000 7fef431877c0 -1 osd.2 62 log_to_monitors true 2024-04-16T11:47:50.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:50 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:51.568 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:51 smithi044 ceph-mon[93020]: pgmap v117: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 836 B/s, 0 objects/s recovering 2024-04-16T11:47:51.568 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:51 smithi044 ceph-mon[93020]: from='osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T11:47:51.568 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 11:47:51 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[113023]: 2024-04-16T11:47:51.270+0000 7fef32600700 -1 osd.2 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:47:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:51 smithi045 ceph-mon[81423]: pgmap v117: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 836 B/s, 0 objects/s recovering 2024-04-16T11:47:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:51 smithi045 ceph-mon[81423]: from='osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T11:47:52.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:52 smithi045 ceph-mon[81423]: from='osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T11:47:52.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:52 smithi045 ceph-mon[81423]: osdmap e68: 6 total, 5 up, 6 in 2024-04-16T11:47:52.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:52 smithi045 ceph-mon[81423]: from='osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:47:52.715 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:52 smithi044 ceph-mon[93020]: from='osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T11:47:52.715 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:52 smithi044 ceph-mon[93020]: osdmap e68: 6 total, 5 up, 6 in 2024-04-16T11:47:52.715 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:52 smithi044 ceph-mon[93020]: from='osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-16T11:47:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:53 smithi045 ceph-mon[81423]: pgmap v119: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 748 B/s, 0 objects/s recovering 2024-04-16T11:47:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:53 smithi045 ceph-mon[81423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:47:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:53 smithi045 ceph-mon[81423]: osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489] boot 2024-04-16T11:47:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:53 smithi045 ceph-mon[81423]: osdmap e69: 6 total, 6 up, 6 in 2024-04-16T11:47:53.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:47:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:53 smithi044 ceph-mon[93020]: pgmap v119: 65 pgs: 4 active+clean+remapped, 6 active+undersized+degraded, 22 active+undersized, 33 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%); 748 B/s, 0 objects/s recovering 2024-04-16T11:47:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:53 smithi044 ceph-mon[93020]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:47:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:53 smithi044 ceph-mon[93020]: osd.2 [v2:172.21.15.44:6818/4137440489,v1:172.21.15.44:6819/4137440489] boot 2024-04-16T11:47:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:53 smithi044 ceph-mon[93020]: osdmap e69: 6 total, 6 up, 6 in 2024-04-16T11:47:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T11:47:54.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:54 smithi045 ceph-mon[81423]: osdmap e70: 6 total, 6 up, 6 in 2024-04-16T11:47:54.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:54.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:54.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:54 smithi045 ceph-mon[81423]: osdmap e71: 6 total, 6 up, 6 in 2024-04-16T11:47:54.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:54 smithi044 ceph-mon[93020]: osdmap e70: 6 total, 6 up, 6 in 2024-04-16T11:47:54.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:54.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:54.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:54 smithi044 ceph-mon[93020]: osdmap e71: 6 total, 6 up, 6 in 2024-04-16T11:47:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:55 smithi044 ceph-mon[93020]: pgmap v122: 65 pgs: 4 active+clean+remapped, 5 active+undersized+degraded, 18 active+undersized, 38 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:47:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:55 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:55 smithi045 ceph-mon[81423]: pgmap v122: 65 pgs: 4 active+clean+remapped, 5 active+undersized+degraded, 18 active+undersized, 38 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:47:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:55 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:47:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:47:56.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:56 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:56.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:56 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:56.905 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:56.905 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:47:57.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:57 smithi044 ceph-mon[93020]: pgmap v124: 65 pgs: 3 active+clean+remapped, 2 active+undersized+degraded, 7 active+undersized, 53 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T11:47:57.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:57 smithi045 ceph-mon[81423]: pgmap v124: 65 pgs: 3 active+clean+remapped, 2 active+undersized+degraded, 7 active+undersized, 53 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T11:47:58.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:58 smithi044 ceph-mon[93020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/72 objects degraded (6.944%), 2 pgs degraded) 2024-04-16T11:47:58.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:58 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:47:59.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:58 smithi045 ceph-mon[81423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/72 objects degraded (6.944%), 2 pgs degraded) 2024-04-16T11:47:59.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:58 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:47:59.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 298, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114961 2024-04-16T11:47:59.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 298, success, outsize: 120 2024-04-16T11:47:59.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 300, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115048 2024-04-16T11:47:59.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 300, success, outsize: 120 2024-04-16T11:47:59.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 302, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115138 2024-04-16T11:47:59.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 302, success, outsize: 120 2024-04-16T11:47:59.746 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 304, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115234 2024-04-16T11:47:59.746 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 304, success, outsize: 120 2024-04-16T11:47:59.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 306, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115327 2024-04-16T11:47:59.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 306, success, outsize: 120 2024-04-16T11:47:59.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 308, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115414 2024-04-16T11:47:59.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 308, success, outsize: 120 2024-04-16T11:47:59.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 310, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115504 2024-04-16T11:47:59.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 310, success, outsize: 120 2024-04-16T11:47:59.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 312, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115600 2024-04-16T11:47:59.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 312, success, outsize: 120 2024-04-16T11:47:59.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 314, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115693 2024-04-16T11:47:59.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 314, success, outsize: 120 2024-04-16T11:47:59.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 316, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115780 2024-04-16T11:47:59.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 316, success, outsize: 120 2024-04-16T11:47:59.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 318, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115870 2024-04-16T11:47:59.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 318, success, outsize: 120 2024-04-16T11:47:59.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 320, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115966 2024-04-16T11:47:59.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 320, success, outsize: 120 2024-04-16T11:47:59.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:47:59 smithi044 ceph-mon[93020]: pgmap v125: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:47:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 322, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116059 2024-04-16T11:47:59.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 322, success, outsize: 120 2024-04-16T11:48:00.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 324, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116146 2024-04-16T11:48:00.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 324, success, outsize: 120 2024-04-16T11:48:00.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 326, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116236 2024-04-16T11:48:00.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 326, success, outsize: 120 2024-04-16T11:48:00.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 328, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116332 2024-04-16T11:48:00.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 328, success, outsize: 120 2024-04-16T11:48:00.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:47:59 smithi045 ceph-mon[81423]: pgmap v125: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:00.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 330, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116425 2024-04-16T11:48:00.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 330, success, outsize: 120 2024-04-16T11:48:00.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 332, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116512 2024-04-16T11:48:00.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 332, success, outsize: 120 2024-04-16T11:48:00.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 334, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116602 2024-04-16T11:48:00.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 334, success, outsize: 120 2024-04-16T11:48:00.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 336, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116698 2024-04-16T11:48:00.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 336, success, outsize: 120 2024-04-16T11:48:00.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 338, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116794 2024-04-16T11:48:00.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 338, success, outsize: 120 2024-04-16T11:48:00.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 340, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116881 2024-04-16T11:48:00.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 340, success, outsize: 120 2024-04-16T11:48:00.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 342, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116971 2024-04-16T11:48:00.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 342, success, outsize: 120 2024-04-16T11:48:00.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 344, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117067 2024-04-16T11:48:00.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 344, success, outsize: 120 2024-04-16T11:48:00.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 346, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117160 2024-04-16T11:48:00.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 346, success, outsize: 120 2024-04-16T11:48:00.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 348, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117248 2024-04-16T11:48:00.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 348, success, outsize: 120 2024-04-16T11:48:00.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 350, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117339 2024-04-16T11:48:00.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 350, success, outsize: 120 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 -- 172.21.15.44:0/3371032593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc0709a0 msgr2=0x7f14fc070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 --2- 172.21.15.44:0/3371032593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc0709a0 0x7f14fc070e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f14e8004940 tx=0x7f14e8009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 -- 172.21.15.44:0/3371032593 shutdown_connections 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 --2- 172.21.15.44:0/3371032593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc0709a0 0x7f14fc070e10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 --2- 172.21.15.44:0/3371032593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14fc071f50 0x7f14fc070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 -- 172.21.15.44:0/3371032593 >> 172.21.15.44:0/3371032593 conn(0x7f14fc0f8530 msgr2=0x7f14fc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 -- 172.21.15.44:0/3371032593 shutdown_connections 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.324+0000 7f1501600700 1 -- 172.21.15.44:0/3371032593 wait complete. 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.325+0000 7f1501600700 1 Processor -- start 2024-04-16T11:48:00.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.325+0000 7f1501600700 1 -- start start 2024-04-16T11:48:00.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f1501600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14fc0709a0 0x7f14fc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f1501600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 0x7f14fc114570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f1501600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc114b70 con 0x7f14fc071f50 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f1501600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc1153c0 con 0x7f14fc0709a0 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f14fa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 0x7f14fc114570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f14fa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 0x7f14fc114570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55986/0 (socket says 172.21.15.44:55986) 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f14fa000700 1 -- 172.21.15.44:0/2212104400 learned_addr learned my addr 172.21.15.44:0/2212104400 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.326+0000 7f14faa00700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14fc0709a0 0x7f14fc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f14fa000700 1 -- 172.21.15.44:0/2212104400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14fc0709a0 msgr2=0x7f14fc114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f14fa000700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14fc0709a0 0x7f14fc114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f14fa000700 1 -- 172.21.15.44:0/2212104400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8005040 con 0x7f14fc071f50 2024-04-16T11:48:00.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f14fa000700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 0x7f14fc114570 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009150 tx=0x7f14e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:00.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14e800e040 con 0x7f14fc071f50 2024-04-16T11:48:00.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14e800bbf0 con 0x7f14fc071f50 2024-04-16T11:48:00.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.327+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc115640 con 0x7f14fc071f50 2024-04-16T11:48:00.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.328+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc115b80 con 0x7f14fc071f50 2024-04-16T11:48:00.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.328+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14e801c630 con 0x7f14fc071f50 2024-04-16T11:48:00.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.329+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc101190 con 0x7f14fc071f50 2024-04-16T11:48:00.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.331+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f14e800bd60 con 0x7f14fc071f50 2024-04-16T11:48:00.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.332+0000 7f14f2a00700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f14e006ec10 0x7f14e00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:00.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.332+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f14e808fe30 con 0x7f14fc071f50 2024-04-16T11:48:00.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.332+0000 7f14faa00700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f14e006ec10 0x7f14e00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:00.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.333+0000 7f14faa00700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f14e006ec10 0x7f14e00710c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007fd0 tx=0x7f14ec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:00.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 352, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117453 2024-04-16T11:48:00.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.338+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f14e805c5f0 con 0x7f14fc071f50 2024-04-16T11:48:00.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 352, success, outsize: 120 2024-04-16T11:48:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.608+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14fc107d50 con 0x7f14e006ec10 2024-04-16T11:48:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.610+0000 7f14f2a00700 1 -- 172.21.15.44:0/2212104400 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f14fc107d50 con 0x7f14e006ec10 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f14e006ec10 msgr2=0x7f14e00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f14e006ec10 0x7f14e00710c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007fd0 tx=0x7f14ec009040 comp rx=0 tx=0).stop 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 msgr2=0x7f14fc114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 0x7f14fc114570 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009150 tx=0x7f14e8009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 shutdown_connections 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f14e006ec10 0x7f14e00710c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14fc0709a0 0x7f14fc114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 --2- 172.21.15.44:0/2212104400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14fc071f50 0x7f14fc114570 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.613+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 >> 172.21.15.44:0/2212104400 conn(0x7f14fc0f8530 msgr2=0x7f14fc0f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:00.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.614+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 shutdown_connections 2024-04-16T11:48:00.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.614+0000 7f1501600700 1 -- 172.21.15.44:0/2212104400 wait complete. 2024-04-16T11:48:00.630 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.776+0000 7f92b7200700 1 -- 172.21.15.44:0/111810223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b0071f50 msgr2=0x7f92b0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.776+0000 7f92b7200700 1 --2- 172.21.15.44:0/111810223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b0071f50 0x7f92b0070460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f92a0004940 tx=0x7f92a0009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.777+0000 7f92b7200700 1 -- 172.21.15.44:0/111810223 shutdown_connections 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.777+0000 7f92b7200700 1 --2- 172.21.15.44:0/111810223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92b00709a0 0x7f92b0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.777+0000 7f92b7200700 1 --2- 172.21.15.44:0/111810223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b0071f50 0x7f92b0070460 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.777+0000 7f92b7200700 1 -- 172.21.15.44:0/111810223 >> 172.21.15.44:0/111810223 conn(0x7f92b00f8530 msgr2=0x7f92b00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.777+0000 7f92b7200700 1 -- 172.21.15.44:0/111810223 shutdown_connections 2024-04-16T11:48:00.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.777+0000 7f92b7200700 1 -- 172.21.15.44:0/111810223 wait complete. 2024-04-16T11:48:00.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.778+0000 7f92b7200700 1 Processor -- start 2024-04-16T11:48:00.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.778+0000 7f92b7200700 1 -- start start 2024-04-16T11:48:00.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.778+0000 7f92b7200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 0x7f92b010fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:00.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.778+0000 7f92b7200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92b0071f50 0x7f92b0110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:00.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.778+0000 7f92b7200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b0110870 con 0x7f92b00709a0 2024-04-16T11:48:00.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.778+0000 7f92b7200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b0110fb0 con 0x7f92b0071f50 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92afe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 0x7f92b010fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92afe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 0x7f92b010fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55990/0 (socket says 172.21.15.44:55990) 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92afe00700 1 -- 172.21.15.44:0/3906290350 learned_addr learned my addr 172.21.15.44:0/3906290350 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92af400700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92b0071f50 0x7f92b0110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92afe00700 1 -- 172.21.15.44:0/3906290350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92b0071f50 msgr2=0x7f92b0110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92afe00700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92b0071f50 0x7f92b0110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:00.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.779+0000 7f92afe00700 1 -- 172.21.15.44:0/3906290350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0005040 con 0x7f92b00709a0 2024-04-16T11:48:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.780+0000 7f92afe00700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 0x7f92b010fd30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92a000b970 tx=0x7f92a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.780+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92a000e040 con 0x7f92b00709a0 2024-04-16T11:48:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.780+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92a0004290 con 0x7f92b00709a0 2024-04-16T11:48:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.780+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92a001c630 con 0x7f92b00709a0 2024-04-16T11:48:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.780+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b0111230 con 0x7f92b00709a0 2024-04-16T11:48:00.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.780+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b01117f0 con 0x7f92b00709a0 2024-04-16T11:48:00.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.781+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b005f5f0 con 0x7f92b00709a0 2024-04-16T11:48:00.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.782+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f92a000bbf0 con 0x7f92b00709a0 2024-04-16T11:48:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.783+0000 7f92a7e00700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f929406ec10 0x7f92940710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.783+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f92a008fcc0 con 0x7f92b00709a0 2024-04-16T11:48:00.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.784+0000 7f92af400700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f929406ec10 0x7f92940710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:00.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.784+0000 7f92af400700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f929406ec10 0x7f92940710c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f929c004990 tx=0x7f929c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:00.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:00.788+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f92a0095020 con 0x7f92b00709a0 2024-04-16T11:48:01.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.046+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92b0111b70 con 0x7f929406ec10 2024-04-16T11:48:01.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.048+0000 7f92a7e00700 1 -- 172.21.15.44:0/3906290350 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f92b0111b70 con 0x7f929406ec10 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.051+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f929406ec10 msgr2=0x7f92940710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.051+0000 7f92b7200700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f929406ec10 0x7f92940710c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f929c004990 tx=0x7f929c009ee0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.051+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 msgr2=0x7f92b010fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.051+0000 7f92b7200700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 0x7f92b010fd30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92a000b970 tx=0x7f92a0009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 shutdown_connections 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f929406ec10 0x7f92940710c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b00709a0 0x7f92b010fd30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 --2- 172.21.15.44:0/3906290350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92b0071f50 0x7f92b0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 >> 172.21.15.44:0/3906290350 conn(0x7f92b00f8530 msgr2=0x7f92b00f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 shutdown_connections 2024-04-16T11:48:01.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.052+0000 7f92b7200700 1 -- 172.21.15.44:0/3906290350 wait complete. 2024-04-16T11:48:01.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.224+0000 7fd83f200700 1 -- 172.21.15.44:0/2435658611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 msgr2=0x7fd838103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.224+0000 7fd83f200700 1 --2- 172.21.15.44:0/2435658611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838103810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd828004940 tx=0x7fd828009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:01.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.224+0000 7fd83f200700 1 -- 172.21.15.44:0/2435658611 shutdown_connections 2024-04-16T11:48:01.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.224+0000 7fd83f200700 1 --2- 172.21.15.44:0/2435658611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381046b0 0x7fd838104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.224+0000 7fd83f200700 1 --2- 172.21.15.44:0/2435658611 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838103810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.224+0000 7fd83f200700 1 -- 172.21.15.44:0/2435658611 >> 172.21.15.44:0/2435658611 conn(0x7fd8380725e0 msgr2=0x7fd838074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:01.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.225+0000 7fd83f200700 1 -- 172.21.15.44:0/2435658611 shutdown_connections 2024-04-16T11:48:01.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.225+0000 7fd83f200700 1 -- 172.21.15.44:0/2435658611 wait complete. 2024-04-16T11:48:01.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.226+0000 7fd83f200700 1 Processor -- start 2024-04-16T11:48:01.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.226+0000 7fd83f200700 1 -- start start 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd83f200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd83f200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381046b0 0x7fd838114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd83f200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838114a60 con 0x7fd838103400 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd83f200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838114bd0 con 0x7fd8381046b0 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd837e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd837e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56002/0 (socket says 172.21.15.44:56002) 2024-04-16T11:48:01.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.227+0000 7fd837e00700 1 -- 172.21.15.44:0/2153689874 learned_addr learned my addr 172.21.15.44:0/2153689874 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd837400700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381046b0 0x7fd838114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd837e00700 1 -- 172.21.15.44:0/2153689874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381046b0 msgr2=0x7fd838114460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd837e00700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381046b0 0x7fd838114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd837e00700 1 -- 172.21.15.44:0/2153689874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd828005040 con 0x7fd838103400 2024-04-16T11:48:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd837e00700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838113f20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd828009150 tx=0x7fd828009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd82800e040 con 0x7fd838103400 2024-04-16T11:48:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd82800bbf0 con 0x7fd838103400 2024-04-16T11:48:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd82801c630 con 0x7fd838103400 2024-04-16T11:48:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.228+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8381195d0 con 0x7fd838103400 2024-04-16T11:48:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.229+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd838119b90 con 0x7fd838103400 2024-04-16T11:48:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.230+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83804c710 con 0x7fd838103400 2024-04-16T11:48:01.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.231+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd82800bd60 con 0x7fd838103400 2024-04-16T11:48:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.232+0000 7fd82fe00700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd81c06ec00 0x7fd81c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.232+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fd82808fd30 con 0x7fd838103400 2024-04-16T11:48:01.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.232+0000 7fd837400700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd81c06ec00 0x7fd81c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:01.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.233+0000 7fd837400700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd81c06ec00 0x7fd81c0710b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd824004d60 tx=0x7fd82400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:01.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.238+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd82805c570 con 0x7fd838103400 2024-04-16T11:48:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.506+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8381089a0 con 0x7fd81c06ec00 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.523+0000 7fd82fe00700 1 -- 172.21.15.44:0/2153689874 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fd8381089a0 con 0x7fd81c06ec00 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (5m) 8s ago 15m 13.2M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (116s) 8s ago 15m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (108s) 103s ago 14m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (5m) 8s ago 15m 74.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (8m) 8s ago 8m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (8m) 8s ago 8m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (8m) 103s ago 8m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:48:01.521 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (8m) 103s ago 8m 13.4M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (4m) 8s ago 17m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (3m) 103s ago 14m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (3m) 8s ago 17m 46.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (2m) 103s ago 14m 31.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (5m) 8s ago 15m 7880k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (4m) 103s ago 14m 8200k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (77s) 8s ago 13m 50.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (46s) 8s ago 12m 42.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (16s) 8s ago 11m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (11m) 103s ago 11m 46.4M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4e456b530904 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (10m) 103s ago 10m 44.8M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (10m) 103s ago 10m 46.6M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:48:01.522 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (4m) 8s ago 14m 56.6M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:48:01.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd81c06ec00 msgr2=0x7fd81c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd81c06ec00 0x7fd81c0710b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd824004d60 tx=0x7fd82400b040 comp rx=0 tx=0).stop 2024-04-16T11:48:01.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 msgr2=0x7fd838113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838113f20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd828009150 tx=0x7fd828009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 shutdown_connections 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd81c06ec00 0x7fd81c0710b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd838103400 0x7fd838113f20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 --2- 172.21.15.44:0/2153689874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381046b0 0x7fd838114460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 >> 172.21.15.44:0/2153689874 conn(0x7fd8380725e0 msgr2=0x7fd838073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 shutdown_connections 2024-04-16T11:48:01.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.526+0000 7fd83f200700 1 -- 172.21.15.44:0/2153689874 wait complete. 2024-04-16T11:48:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 -- 172.21.15.44:0/3801645685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280662f0 msgr2=0x7f2e280ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 --2- 172.21.15.44:0/3801645685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280662f0 0x7f2e280ff800 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2e18004940 tx=0x7f2e18009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 -- 172.21.15.44:0/3801645685 shutdown_connections 2024-04-16T11:48:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 --2- 172.21.15.44:0/3801645685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e280ffd40 0x7f2e28102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 --2- 172.21.15.44:0/3801645685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280662f0 0x7f2e280ff800 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 -- 172.21.15.44:0/3801645685 >> 172.21.15.44:0/3801645685 conn(0x7f2e280f7270 msgr2=0x7f2e280f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 -- 172.21.15.44:0/3801645685 shutdown_connections 2024-04-16T11:48:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.682+0000 7f2e27200700 1 -- 172.21.15.44:0/3801645685 wait complete. 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.683+0000 7f2e27200700 1 Processor -- start 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.683+0000 7f2e27200700 1 -- start start 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e27200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e280662f0 0x7f2e2810fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e27200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 0x7f2e2810ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e27200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281105b0 con 0x7f2e280ffd40 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e27200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e28110db0 con 0x7f2e280662f0 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 0x7f2e2810ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 0x7f2e2810ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36368/0 (socket says 172.21.15.44:36368) 2024-04-16T11:48:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25400700 1 -- 172.21.15.44:0/803425715 learned_addr learned my addr 172.21.15.44:0/803425715 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25e00700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e280662f0 0x7f2e2810fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25400700 1 -- 172.21.15.44:0/803425715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e280662f0 msgr2=0x7f2e2810fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25400700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e280662f0 0x7f2e2810fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.684+0000 7f2e25400700 1 -- 172.21.15.44:0/803425715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18005040 con 0x7f2e280ffd40 2024-04-16T11:48:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.685+0000 7f2e25400700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 0x7f2e2810ffb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400eb40 tx=0x7f2e1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.685+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e14007cc0 con 0x7f2e280ffd40 2024-04-16T11:48:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.685+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e1400f040 con 0x7f2e280ffd40 2024-04-16T11:48:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.685+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e14009620 con 0x7f2e280ffd40 2024-04-16T11:48:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.685+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e28111030 con 0x7f2e280ffd40 2024-04-16T11:48:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.685+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e28111520 con 0x7f2e280ffd40 2024-04-16T11:48:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.686+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2805f5f0 con 0x7f2e280ffd40 2024-04-16T11:48:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.688+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e14009780 con 0x7f2e280ffd40 2024-04-16T11:48:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.689+0000 7f2e1e000700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2e0c06ec10 0x7f2e0c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.689+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f2e1408e6c0 con 0x7f2e280ffd40 2024-04-16T11:48:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.689+0000 7f2e25e00700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2e0c06ec10 0x7f2e0c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.690+0000 7f2e25e00700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2e0c06ec10 0x7f2e0c0710c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e18013870 tx=0x7f2e18009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:01.694+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2e1405aeb0 con 0x7f2e280ffd40 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: pgmap v126: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:48:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:02.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.025+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2e2804bed0 con 0x7f2e280ffd40 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.026+0000 7f2e1e000700 1 -- 172.21.15.44:0/803425715 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f2e1405aa40 con 0x7f2e280ffd40 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:48:02.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 7, 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 7 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:48:02.025 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:48:02.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2e0c06ec10 msgr2=0x7f2e0c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2e0c06ec10 0x7f2e0c0710c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e18013870 tx=0x7f2e18009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 msgr2=0x7f2e2810ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 0x7f2e2810ffb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400eb40 tx=0x7f2e1400b040 comp rx=0 tx=0).stop 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 shutdown_connections 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2e0c06ec10 0x7f2e0c0710c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e280662f0 0x7f2e2810fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 --2- 172.21.15.44:0/803425715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2e280ffd40 0x7f2e2810ffb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.028+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 >> 172.21.15.44:0/803425715 conn(0x7f2e280f7270 msgr2=0x7f2e280fb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.029+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 shutdown_connections 2024-04-16T11:48:02.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.029+0000 7f2e27200700 1 -- 172.21.15.44:0/803425715 wait complete. 2024-04-16T11:48:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: pgmap v126: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:48:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:02.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.182+0000 7ff919e00700 1 -- 172.21.15.44:0/1304526654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9140fc0b0 msgr2=0x7ff9140fc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.182+0000 7ff919e00700 1 --2- 172.21.15.44:0/1304526654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9140fc0b0 0x7ff9140fc770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc004a50 tx=0x7ff8fc00af40 comp rx=0 tx=0).stop 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.182+0000 7ff919e00700 1 -- 172.21.15.44:0/1304526654 shutdown_connections 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.183+0000 7ff919e00700 1 --2- 172.21.15.44:0/1304526654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9140fc0b0 0x7ff9140fc770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.183+0000 7ff919e00700 1 --2- 172.21.15.44:0/1304526654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9140fb760 0x7ff9140fbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.183+0000 7ff919e00700 1 -- 172.21.15.44:0/1304526654 >> 172.21.15.44:0/1304526654 conn(0x7ff9140f6770 msgr2=0x7ff9140f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.183+0000 7ff919e00700 1 -- 172.21.15.44:0/1304526654 shutdown_connections 2024-04-16T11:48:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.183+0000 7ff919e00700 1 -- 172.21.15.44:0/1304526654 wait complete. 2024-04-16T11:48:02.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.183+0000 7ff919e00700 1 Processor -- start 2024-04-16T11:48:02.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff919e00700 1 -- start start 2024-04-16T11:48:02.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff919e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9140fb760 0x7ff91410d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:02.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff919e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 0x7ff914112e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:02.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff919e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91410e300 con 0x7ff91410de40 2024-04-16T11:48:02.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff919e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91410e440 con 0x7ff9140fb760 2024-04-16T11:48:02.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff90a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 0x7ff914112e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:02.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff90a000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 0x7ff914112e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36384/0 (socket says 172.21.15.44:36384) 2024-04-16T11:48:02.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.184+0000 7ff90a000700 1 -- 172.21.15.44:0/1900113815 learned_addr learned my addr 172.21.15.44:0/1900113815 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:02.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.185+0000 7ff912a00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9140fb760 0x7ff91410d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.185+0000 7ff90a000700 1 -- 172.21.15.44:0/1900113815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9140fb760 msgr2=0x7ff91410d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.185+0000 7ff90a000700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9140fb760 0x7ff91410d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.185+0000 7ff90a000700 1 -- 172.21.15.44:0/1900113815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc005040 con 0x7ff91410de40 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.185+0000 7ff90a000700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 0x7ff914112e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc004710 tx=0x7ff8fc00af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.186+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8fc00f050 con 0x7ff91410de40 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.186+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8fc01d950 con 0x7ff91410de40 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.186+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9141133b0 con 0x7ff91410de40 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.186+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8fc0149e0 con 0x7ff91410de40 2024-04-16T11:48:02.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.186+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff914113830 con 0x7ff91410de40 2024-04-16T11:48:02.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.187+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91404c710 con 0x7ff91410de40 2024-04-16T11:48:02.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.188+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff8fc01dac0 con 0x7ff91410de40 2024-04-16T11:48:02.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.189+0000 7ff90b400700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff8f806ec00 0x7ff8f80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:02.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.189+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7ff8fc0907d0 con 0x7ff91410de40 2024-04-16T11:48:02.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.190+0000 7ff912a00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff8f806ec00 0x7ff8f80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:02.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.190+0000 7ff912a00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff8f806ec00 0x7ff8f80710b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff904004550 tx=0x7ff904009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:02.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.194+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff8fc05d010 con 0x7ff91410de40 2024-04-16T11:48:02.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.480+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff9141036d0 con 0x7ff91410de40 2024-04-16T11:48:02.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.481+0000 7ff90b400700 1 -- 172.21.15.44:0/1900113815 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7ff8fc05cba0 con 0x7ff91410de40 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:48:02.481 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:48:02.482 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:02.483 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:02.484 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:02.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff8f806ec00 msgr2=0x7ff8f80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff8f806ec00 0x7ff8f80710b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff904004550 tx=0x7ff904009040 comp rx=0 tx=0).stop 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 msgr2=0x7ff914112e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 0x7ff914112e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc004710 tx=0x7ff8fc00af40 comp rx=0 tx=0).stop 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 shutdown_connections 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff8f806ec00 0x7ff8f80710b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9140fb760 0x7ff91410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 --2- 172.21.15.44:0/1900113815 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff91410de40 0x7ff914112e70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 >> 172.21.15.44:0/1900113815 conn(0x7ff9140f6770 msgr2=0x7ff9140f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 shutdown_connections 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.484+0000 7ff919e00700 1 -- 172.21.15.44:0/1900113815 wait complete. 2024-04-16T11:48:02.485 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:48:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 -- 172.21.15.44:0/1237718897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 msgr2=0x7fb7341023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 --2- 172.21.15.44:0/1237718897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341023d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb720004940 tx=0x7fb720009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:02.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 -- 172.21.15.44:0/1237718897 shutdown_connections 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 --2- 172.21.15.44:0/1237718897 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341023d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 --2- 172.21.15.44:0/1237718897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb734066b80 0x7fb7340ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 -- 172.21.15.44:0/1237718897 >> 172.21.15.44:0/1237718897 conn(0x7fb7340f74e0 msgr2=0x7fb7340f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 -- 172.21.15.44:0/1237718897 shutdown_connections 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.649+0000 7fb73ae00700 1 -- 172.21.15.44:0/1237718897 wait complete. 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.650+0000 7fb73ae00700 1 Processor -- start 2024-04-16T11:48:02.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.651+0000 7fb73ae00700 1 -- start start 2024-04-16T11:48:02.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.651+0000 7fb73ae00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb734066b80 0x7fb73410fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.651+0000 7fb73ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341101c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.651+0000 7fb73ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7341107c0 con 0x7fb7340fffb0 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.651+0000 7fb73ae00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb734110fc0 con 0x7fb734066b80 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341101c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36406/0 (socket says 172.21.15.44:36406) 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733400700 1 -- 172.21.15.44:0/2595264884 learned_addr learned my addr 172.21.15.44:0/2595264884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:02.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733e00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb734066b80 0x7fb73410fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733400700 1 -- 172.21.15.44:0/2595264884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb734066b80 msgr2=0x7fb73410fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733400700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb734066b80 0x7fb73410fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.652+0000 7fb733400700 1 -- 172.21.15.44:0/2595264884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720005040 con 0x7fb7340fffb0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.653+0000 7fb733400700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341101c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb720013870 tx=0x7fb720009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.653+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb72000e070 con 0x7fb7340fffb0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.653+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb720013de0 con 0x7fb7340fffb0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.653+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb72001cea0 con 0x7fb7340fffb0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.653+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb734111240 con 0x7fb7340fffb0 2024-04-16T11:48:02.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.653+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb734111720 con 0x7fb7340fffb0 2024-04-16T11:48:02.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.654+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73405f5f0 con 0x7fb7340fffb0 2024-04-16T11:48:02.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.656+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb72001c690 con 0x7fb7340fffb0 2024-04-16T11:48:02.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.657+0000 7fb72be00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb71806ec10 0x7fb7180710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:02.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.657+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fb720090070 con 0x7fb7340fffb0 2024-04-16T11:48:02.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.657+0000 7fb733e00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb71806ec10 0x7fb7180710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:02.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.658+0000 7fb733e00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb71806ec10 0x7fb7180710c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb724009b80 tx=0x7fb72400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:02.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.662+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb72005c8b0 con 0x7fb7340fffb0 2024-04-16T11:48:02.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.919+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb734111a50 con 0x7fb71806ec10 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.917 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T11:48:02.918 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T11:48:02.918 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: Upgrade: osd.3 is safe to restart 2024-04-16T11:48:02.918 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/803425715' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.918 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:02 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1900113815' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.921+0000 7fb72be00700 1 -- 172.21.15.44:0/2595264884 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb734111a50 con 0x7fb71806ec10 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:48:02.919 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:48:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:48:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:48:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "9/21 daemons upgraded", 2024-04-16T11:48:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T11:48:02.920 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:48:02.920 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:48:02.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.924+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb71806ec10 msgr2=0x7fb7180710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.924+0000 7fb73ae00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb71806ec10 0x7fb7180710c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb724009b80 tx=0x7fb72400b040 comp rx=0 tx=0).stop 2024-04-16T11:48:02.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.924+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 msgr2=0x7fb7341101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:02.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.924+0000 7fb73ae00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341101c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb720013870 tx=0x7fb720009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 shutdown_connections 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb71806ec10 0x7fb7180710c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb734066b80 0x7fb73410fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 --2- 172.21.15.44:0/2595264884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7340fffb0 0x7fb7341101c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 >> 172.21.15.44:0/2595264884 conn(0x7fb7340f74e0 msgr2=0x7fb7340fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 shutdown_connections 2024-04-16T11:48:02.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:02.925+0000 7fb73ae00700 1 -- 172.21.15.44:0/2595264884 wait complete. 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T11:48:02.932 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T11:48:02.933 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: Upgrade: osd.3 is safe to restart 2024-04-16T11:48:02.933 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/803425715' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:02.933 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:02 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1900113815' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:48:03.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 -- 172.21.15.44:0/3731252587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 msgr2=0x7f77ec070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:03.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 --2- 172.21.15.44:0/3731252587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec070d00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f77d8004940 tx=0x7f77d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:03.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 -- 172.21.15.44:0/3731252587 shutdown_connections 2024-04-16T11:48:03.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 --2- 172.21.15.44:0/3731252587 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec070d00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:03.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 --2- 172.21.15.44:0/3731252587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec071e40 0x7f77ec070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:03.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 -- 172.21.15.44:0/3731252587 >> 172.21.15.44:0/3731252587 conn(0x7f77ec0f8580 msgr2=0x7f77ec0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:03.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 -- 172.21.15.44:0/3731252587 shutdown_connections 2024-04-16T11:48:03.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.076+0000 7f77f1200700 1 -- 172.21.15.44:0/3731252587 wait complete. 2024-04-16T11:48:03.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.077+0000 7f77f1200700 1 Processor -- start 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.077+0000 7f77f1200700 1 -- start start 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77f1200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77f1200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec071e40 0x7f77ec110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77f1200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec110680 con 0x7f77ec070890 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77f1200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec1107f0 con 0x7f77ec071e40 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec10fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36418/0 (socket says 172.21.15.44:36418) 2024-04-16T11:48:03.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9e00700 1 -- 172.21.15.44:0/2874887464 learned_addr learned my addr 172.21.15.44:0/2874887464 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9e00700 1 -- 172.21.15.44:0/2874887464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec071e40 msgr2=0x7f77ec110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9400700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec071e40 0x7f77ec110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9e00700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec071e40 0x7f77ec110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.078+0000 7f77e9e00700 1 -- 172.21.15.44:0/2874887464 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8005040 con 0x7f77ec070890 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.079+0000 7f77e9e00700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec10fb40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f77dc002a80 tx=0x7f77dc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.079+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77dc007cc0 con 0x7f77ec070890 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.079+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77dc00f040 con 0x7f77ec070890 2024-04-16T11:48:03.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.079+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec115250 con 0x7f77ec070890 2024-04-16T11:48:03.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.079+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77dc00dad0 con 0x7f77ec070890 2024-04-16T11:48:03.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.079+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec115810 con 0x7f77ec070890 2024-04-16T11:48:03.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.080+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec04c710 con 0x7f77ec070890 2024-04-16T11:48:03.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.081+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77dc00dc30 con 0x7f77ec070890 2024-04-16T11:48:03.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.082+0000 7f77e2000700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec00 0x7f77d00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:03.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.082+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f77dc08e7d0 con 0x7f77ec070890 2024-04-16T11:48:03.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.082+0000 7f77e9400700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec00 0x7f77d00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:03.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.083+0000 7f77e9400700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec00 0x7f77d00710b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b970 tx=0x7f77d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:03.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.087+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f77dc05af10 con 0x7f77ec070890 2024-04-16T11:48:03.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.411+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f77ec115c90 con 0x7f77ec070890 2024-04-16T11:48:03.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.412+0000 7f77e2000700 1 -- 172.21.15.44:0/2874887464 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f77dc05aaa0 con 0x7f77ec070890 2024-04-16T11:48:03.410 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:48:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec00 msgr2=0x7f77d00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec00 0x7f77d00710b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b970 tx=0x7f77d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 msgr2=0x7f77ec10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec10fb40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f77dc002a80 tx=0x7f77dc004600 comp rx=0 tx=0).stop 2024-04-16T11:48:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 shutdown_connections 2024-04-16T11:48:03.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f77d006ec00 0x7f77d00710b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:03.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77ec070890 0x7f77ec10fb40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:03.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 --2- 172.21.15.44:0/2874887464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77ec071e40 0x7f77ec110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:03.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 >> 172.21.15.44:0/2874887464 conn(0x7f77ec0f8580 msgr2=0x7f77ec0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:03.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 shutdown_connections 2024-04-16T11:48:03.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:03.415+0000 7f77f1200700 1 -- 172.21.15.44:0/2874887464 wait complete. 2024-04-16T11:48:03.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:03 smithi044 ceph-mon[93020]: pgmap v127: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:03.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:03 smithi044 ceph-mon[93020]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:03.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:03 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2874887464' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:48:04.029 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:03 smithi045 ceph-mon[81423]: pgmap v127: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:04.029 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:03 smithi045 ceph-mon[81423]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:04.029 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:03 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2874887464' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:48:05.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:05 smithi045 ceph-mon[81423]: Upgrade: Updating osd.3 2024-04-16T11:48:05.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:05.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T11:48:05.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:05.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:05 smithi045 ceph-mon[81423]: Deploying daemon osd.3 on smithi045 2024-04-16T11:48:05.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:05 smithi045 ceph-mon[81423]: pgmap v128: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:05 smithi044 ceph-mon[93020]: Upgrade: Updating osd.3 2024-04-16T11:48:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T11:48:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:05 smithi044 ceph-mon[93020]: Deploying daemon osd.3 on smithi045 2024-04-16T11:48:05.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:05 smithi044 ceph-mon[93020]: pgmap v128: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:06.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:06 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:06.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:06 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:07.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:07 smithi045 ceph-mon[81423]: pgmap v129: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:07.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:07 smithi044 ceph-mon[93020]: pgmap v129: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:09.079 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:08 smithi045 systemd[1]: Stopping Ceph osd.3 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:48:09.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:09 smithi045 ceph-mon[81423]: pgmap v130: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:09.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:09 smithi045 ceph-mon[81423]: osd.3 marked itself down and dead 2024-04-16T11:48:09.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:09 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[41815]: 2024-04-16T11:48:09.224+0000 7f3c44200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:48:09.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:09 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[41815]: 2024-04-16T11:48:09.224+0000 7f3c44200700 -1 osd.3 71 *** Got signal Terminated *** 2024-04-16T11:48:09.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:09 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[41815]: 2024-04-16T11:48:09.224+0000 7f3c44200700 -1 osd.3 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T11:48:09.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:09 smithi044 ceph-mon[93020]: pgmap v130: 65 pgs: 65 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:09.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:09 smithi044 ceph-mon[93020]: osd.3 marked itself down and dead 2024-04-16T11:48:10.203 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:09 smithi045 podman[90283]: 2024-04-16 11:48:09.946952016 +0000 UTC m=+0.844054014 container died 4e456b53090421ac357183a6329441574fa1df4c736e5272fd773bc13a0c999a (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:48:10.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:10 smithi045 ceph-mon[81423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:48:10.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:10 smithi045 ceph-mon[81423]: osdmap e72: 6 total, 5 up, 6 in 2024-04-16T11:48:10.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:10.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:10.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:10 smithi045 ceph-mon[81423]: osdmap e73: 6 total, 5 up, 6 in 2024-04-16T11:48:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:10 smithi044 ceph-mon[93020]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:48:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:10 smithi044 ceph-mon[93020]: osdmap e72: 6 total, 5 up, 6 in 2024-04-16T11:48:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:10.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:10 smithi044 ceph-mon[93020]: osdmap e73: 6 total, 5 up, 6 in 2024-04-16T11:48:10.829 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:10 smithi045 podman[90283]: 2024-04-16 11:48:10.502825641 +0000 UTC m=+1.399927654 container cleanup 4e456b53090421ac357183a6329441574fa1df4c736e5272fd773bc13a0c999a (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409) 2024-04-16T11:48:10.829 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:10 smithi045 bash[90283]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3 2024-04-16T11:48:10.830 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:10 smithi045 podman[90301]: 2024-04-16 11:48:10.676070404 +0000 UTC m=+0.712107270 container remove 4e456b53090421ac357183a6329441574fa1df4c736e5272fd773bc13a0c999a (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T11:48:11.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:11 smithi045 ceph-mon[81423]: pgmap v132: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:11.654 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:11 smithi045 podman[90391]: 2024-04-16 11:48:11.398538478 +0000 UTC m=+0.036641646 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:48:11.655 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:11 smithi045 podman[90391]: 2024-04-16 11:48:11.50217408 +0000 UTC m=+0.140277273 container create a25baf429460ba7ebbc5a2574f36ad8164dc3b39783273621e7c0c9635f45b9f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-16T11:48:11.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:11 smithi044 ceph-mon[93020]: pgmap v132: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-16T11:48:12.079 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:11 smithi045 podman[90391]: 2024-04-16 11:48:11.781941698 +0000 UTC m=+0.420044924 container init a25baf429460ba7ebbc5a2574f36ad8164dc3b39783273621e7c0c9635f45b9f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-16T11:48:12.080 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:11 smithi045 podman[90391]: 2024-04-16 11:48:11.794160443 +0000 UTC m=+0.432263606 container start a25baf429460ba7ebbc5a2574f36ad8164dc3b39783273621e7c0c9635f45b9f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0) 2024-04-16T11:48:12.080 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:11 smithi045 podman[90391]: 2024-04-16 11:48:11.871565239 +0000 UTC m=+0.509668475 container attach a25baf429460ba7ebbc5a2574f36ad8164dc3b39783273621e7c0c9635f45b9f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-16T11:48:12.537 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:12 smithi045 ceph-mon[81423]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:12.538 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:12 smithi045 podman[90391]: 2024-04-16 11:48:12.257638628 +0000 UTC m=+0.895741871 container died a25baf429460ba7ebbc5a2574f36ad8164dc3b39783273621e7c0c9635f45b9f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-16T11:48:12.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:12 smithi044 ceph-mon[93020]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:13.275 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:12 smithi045 podman[90417]: 2024-04-16 11:48:12.866813143 +0000 UTC m=+0.591325210 container remove a25baf429460ba7ebbc5a2574f36ad8164dc3b39783273621e7c0c9635f45b9f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-04-16T11:48:13.275 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:12 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-16T11:48:13.275 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:12 smithi045 systemd[1]: Stopped Ceph osd.3 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:48:13.275 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:12 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3.service: Consumed 14.343s CPU time. 2024-04-16T11:48:13.579 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:13 smithi045 systemd[1]: Starting Ceph osd.3 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:48:13.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:13 smithi045 ceph-mon[81423]: pgmap v134: 65 pgs: 5 active+undersized+degraded, 8 peering, 3 stale+active+clean, 13 active+undersized, 36 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:48:13.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:13 smithi044 ceph-mon[93020]: pgmap v134: 65 pgs: 5 active+undersized+degraded, 8 peering, 3 stale+active+clean, 13 active+undersized, 36 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:48:13.929 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:13 smithi045 podman[90512]: 2024-04-16 11:48:13.574672217 +0000 UTC m=+0.040698522 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:48:13.929 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:13 smithi045 podman[90512]: 2024-04-16 11:48:13.668224674 +0000 UTC m=+0.134251016 container create 3e7c1928b09d346f75e9c37430fd2accff0eacac73f92e7c72b5c2e9c522ffce (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-16T11:48:13.929 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:13 smithi045 podman[90512]: 2024-04-16 11:48:13.930893447 +0000 UTC m=+0.396919786 container init 3e7c1928b09d346f75e9c37430fd2accff0eacac73f92e7c72b5c2e9c522ffce (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409) 2024-04-16T11:48:14.329 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:13 smithi045 podman[90512]: 2024-04-16 11:48:13.943024571 +0000 UTC m=+0.409050879 container start 3e7c1928b09d346f75e9c37430fd2accff0eacac73f92e7c72b5c2e9c522ffce (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T11:48:14.330 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 podman[90512]: 2024-04-16 11:48:14.012690808 +0000 UTC m=+0.478717200 container attach 3e7c1928b09d346f75e9c37430fd2accff0eacac73f92e7c72b5c2e9c522ffce (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T11:48:14.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 348, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90634 2024-04-16T11:48:14.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 348, success, outsize: 120 2024-04-16T11:48:15.244 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T11:48:15.244 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 bash[90512]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T11:48:15.245 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate[90527]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T11:48:15.246 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:14 smithi045 podman[90512]: 2024-04-16 11:48:14.901066337 +0000 UTC m=+1.367092645 container died 3e7c1928b09d346f75e9c37430fd2accff0eacac73f92e7c72b5c2e9c522ffce (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T11:48:15.555 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:15 smithi045 ceph-mon[81423]: pgmap v135: 65 pgs: 7 active+undersized+degraded, 8 peering, 18 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T11:48:15.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:15 smithi044 ceph-mon[93020]: pgmap v135: 65 pgs: 7 active+undersized+degraded, 8 peering, 18 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T11:48:15.829 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:15 smithi045 podman[90662]: 2024-04-16 11:48:15.557990855 +0000 UTC m=+0.635973284 container remove 3e7c1928b09d346f75e9c37430fd2accff0eacac73f92e7c72b5c2e9c522ffce (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-16T11:48:16.209 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:15 smithi045 podman[90721]: 2024-04-16 11:48:15.847893078 +0000 UTC m=+0.039236333 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:48:16.209 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:15 smithi045 podman[90721]: 2024-04-16 11:48:15.950310344 +0000 UTC m=+0.141653579 container create be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:48:16.579 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:16 smithi045 podman[90721]: 2024-04-16 11:48:16.212266699 +0000 UTC m=+0.403609969 container init be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-16T11:48:16.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:16 smithi045 podman[90721]: 2024-04-16 11:48:16.223886755 +0000 UTC m=+0.415230025 container start be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-16T11:48:16.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:16 smithi045 ceph-osd[90740]: -- 172.21.15.45:0/2435927650 <== mon.1 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c044627680 con 0x55c0445f1000 2024-04-16T11:48:16.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:16 smithi045 bash[90721]: be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 2024-04-16T11:48:16.580 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:16 smithi045 systemd[1]: Started Ceph osd.3 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:48:17.079 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:16 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T11:48:16.792+0000 7f06898f97c0 -1 Falling back to public interface 2024-04-16T11:48:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:17 smithi044 ceph-mon[93020]: pgmap v136: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:17.790 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:17 smithi045 ceph-mon[81423]: pgmap v136: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:17.790 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:17.790 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:17.790 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:19 smithi044 ceph-mon[93020]: pgmap v137: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:19.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:19 smithi045 ceph-mon[81423]: pgmap v137: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:20.829 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:20 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T11:48:20.427+0000 7f06898f97c0 -1 osd.3 71 log_to_monitors true 2024-04-16T11:48:20.829 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 11:48:20 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T11:48:20.505+0000 7f0678c00700 -1 osd.3 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:48:20.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:20 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:20.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:20 smithi045 ceph-mon[81423]: from='osd.3 [v2:172.21.15.45:6800/2814984900,v1:172.21.15.45:6801/2814984900]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:48:20.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:20 smithi045 ceph-mon[81423]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:48:20.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:20 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:20 smithi044 ceph-mon[93020]: from='osd.3 [v2:172.21.15.45:6800/2814984900,v1:172.21.15.45:6801/2814984900]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:48:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:20 smithi044 ceph-mon[93020]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T11:48:21.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: pgmap v138: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T11:48:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: from='osd.3 [v2:172.21.15.45:6800/2814984900,v1:172.21.15.45:6801/2814984900]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: osdmap e74: 6 total, 5 up, 6 in 2024-04-16T11:48:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:21.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: pgmap v138: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 966 MiB used, 535 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: from='osd.3 [v2:172.21.15.45:6800/2814984900,v1:172.21.15.45:6801/2814984900]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: osdmap e74: 6 total, 5 up, 6 in 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:22 smithi045 ceph-mon[81423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:48:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:22 smithi045 ceph-mon[81423]: osd.3 [v2:172.21.15.45:6800/2814984900,v1:172.21.15.45:6801/2814984900] boot 2024-04-16T11:48:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:22 smithi045 ceph-mon[81423]: osdmap e75: 6 total, 6 up, 6 in 2024-04-16T11:48:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:22 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:48:22.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:22 smithi045 ceph-mon[81423]: osdmap e76: 6 total, 6 up, 6 in 2024-04-16T11:48:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:22 smithi044 ceph-mon[93020]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:48:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:22 smithi044 ceph-mon[93020]: osd.3 [v2:172.21.15.45:6800/2814984900,v1:172.21.15.45:6801/2814984900] boot 2024-04-16T11:48:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:22 smithi044 ceph-mon[93020]: osdmap e75: 6 total, 6 up, 6 in 2024-04-16T11:48:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:22 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T11:48:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:22 smithi044 ceph-mon[93020]: osdmap e76: 6 total, 6 up, 6 in 2024-04-16T11:48:23.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:23 smithi045 ceph-mon[81423]: pgmap v141: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:23.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:23 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:23.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:23 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:23.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:23 smithi044 ceph-mon[93020]: pgmap v141: 65 pgs: 10 active+undersized+degraded, 23 active+undersized, 32 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-04-16T11:48:23.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:23 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:23.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:23 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:25.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:25 smithi045 ceph-mon[81423]: pgmap v143: 65 pgs: 9 active+undersized+degraded, 20 active+undersized, 36 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T11:48:25.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:25 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 14/72 objects degraded (19.444%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:25.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:25.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:25 smithi044 ceph-mon[93020]: pgmap v143: 65 pgs: 9 active+undersized+degraded, 20 active+undersized, 36 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T11:48:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:25 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 14/72 objects degraded (19.444%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:27.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 350, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92459 2024-04-16T11:48:27.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 350, success, outsize: 120 2024-04-16T11:48:27.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 352, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92553 2024-04-16T11:48:27.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 352, success, outsize: 120 2024-04-16T11:48:27.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 354, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92650 2024-04-16T11:48:27.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 354, success, outsize: 120 2024-04-16T11:48:27.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92753 2024-04-16T11:48:27.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 356, success, outsize: 120 2024-04-16T11:48:27.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 358, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92853 2024-04-16T11:48:27.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 358, success, outsize: 120 2024-04-16T11:48:27.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 360, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92947 2024-04-16T11:48:27.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 360, success, outsize: 120 2024-04-16T11:48:27.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 362, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93044 2024-04-16T11:48:27.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 362, success, outsize: 120 2024-04-16T11:48:27.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93147 2024-04-16T11:48:27.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 364, success, outsize: 120 2024-04-16T11:48:27.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 366, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93247 2024-04-16T11:48:27.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 366, success, outsize: 120 2024-04-16T11:48:27.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 368, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93341 2024-04-16T11:48:27.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 368, success, outsize: 120 2024-04-16T11:48:27.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 370, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93438 2024-04-16T11:48:27.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 370, success, outsize: 120 2024-04-16T11:48:27.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 372, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93541 2024-04-16T11:48:27.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 372, success, outsize: 120 2024-04-16T11:48:27.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 374, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93641 2024-04-16T11:48:27.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 374, success, outsize: 120 2024-04-16T11:48:27.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 376, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93735 2024-04-16T11:48:27.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 376, success, outsize: 120 2024-04-16T11:48:27.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 378, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93832 2024-04-16T11:48:27.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 378, success, outsize: 120 2024-04-16T11:48:27.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 380, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93935 2024-04-16T11:48:27.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 380, success, outsize: 120 2024-04-16T11:48:27.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 382, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94035 2024-04-16T11:48:27.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 382, success, outsize: 120 2024-04-16T11:48:27.611 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 384, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94129 2024-04-16T11:48:27.611 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 384, success, outsize: 120 2024-04-16T11:48:27.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 386, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94226 2024-04-16T11:48:27.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 386, success, outsize: 120 2024-04-16T11:48:27.669 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 388, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94329 2024-04-16T11:48:27.669 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 388, success, outsize: 120 2024-04-16T11:48:27.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 390, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94429 2024-04-16T11:48:27.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 390, success, outsize: 120 2024-04-16T11:48:27.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 392, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94523 2024-04-16T11:48:27.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 392, success, outsize: 120 2024-04-16T11:48:27.760 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 394, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94620 2024-04-16T11:48:27.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 394, success, outsize: 120 2024-04-16T11:48:27.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 396, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94723 2024-04-16T11:48:27.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 396, success, outsize: 120 2024-04-16T11:48:27.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 398, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94823 2024-04-16T11:48:27.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 398, success, outsize: 120 2024-04-16T11:48:27.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:27 smithi045 ceph-mon[81423]: pgmap v144: 65 pgs: 1 active+undersized+degraded, 2 active+undersized, 62 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 1/72 objects degraded (1.389%) 2024-04-16T11:48:27.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 400, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94917 2024-04-16T11:48:27.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 400, success, outsize: 120 2024-04-16T11:48:27.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 402, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 95014 2024-04-16T11:48:27.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 402, success, outsize: 120 2024-04-16T11:48:27.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 404, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 95117 2024-04-16T11:48:27.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 404, success, outsize: 120 2024-04-16T11:48:27.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:27 smithi044 ceph-mon[93020]: pgmap v144: 65 pgs: 1 active+undersized+degraded, 2 active+undersized, 62 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 1/72 objects degraded (1.389%) 2024-04-16T11:48:28.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:28 smithi044 ceph-mon[93020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/72 objects degraded (1.389%), 1 pg degraded) 2024-04-16T11:48:28.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:28 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:48:29.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:28 smithi045 ceph-mon[81423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/72 objects degraded (1.389%), 1 pg degraded) 2024-04-16T11:48:29.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:28 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: pgmap v145: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:29.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:29.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:29.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T11:48:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: pgmap v145: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:48:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:30.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:30.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:30.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:30.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:30.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T11:48:30.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:30 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T11:48:30.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:30 smithi044 ceph-mon[93020]: Upgrade: osd.4 is safe to restart 2024-04-16T11:48:31.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:30 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T11:48:31.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:30 smithi045 ceph-mon[81423]: Upgrade: osd.4 is safe to restart 2024-04-16T11:48:31.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:31 smithi045 ceph-mon[81423]: pgmap v146: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:31.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:31.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T11:48:31.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:31 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:31.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:31 smithi044 ceph-mon[93020]: pgmap v146: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:31.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:31.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T11:48:31.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:31 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:32.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:32 smithi045 ceph-mon[81423]: Upgrade: Updating osd.4 2024-04-16T11:48:32.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:32 smithi045 ceph-mon[81423]: Deploying daemon osd.4 on smithi045 2024-04-16T11:48:32.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:32 smithi044 ceph-mon[93020]: Upgrade: Updating osd.4 2024-04-16T11:48:32.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:32 smithi044 ceph-mon[93020]: Deploying daemon osd.4 on smithi045 2024-04-16T11:48:33.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 -- 172.21.15.44:0/1970561284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fe040 msgr2=0x7f10f00fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:33.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1970561284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fe040 0x7f10f00fe4b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f10dc004940 tx=0x7f10dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:33.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 -- 172.21.15.44:0/1970561284 shutdown_connections 2024-04-16T11:48:33.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1970561284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fe040 0x7f10f00fe4b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:33.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1970561284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f00fce40 0x7f10f00fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:33.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 -- 172.21.15.44:0/1970561284 >> 172.21.15.44:0/1970561284 conn(0x7f10f00f8530 msgr2=0x7f10f00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:33.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 -- 172.21.15.44:0/1970561284 shutdown_connections 2024-04-16T11:48:33.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.590+0000 7f10f7a00700 1 -- 172.21.15.44:0/1970561284 wait complete. 2024-04-16T11:48:33.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.591+0000 7f10f7a00700 1 Processor -- start 2024-04-16T11:48:33.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.591+0000 7f10f7a00700 1 -- start start 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f7a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 0x7f10f010fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f7a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f00fe040 0x7f10f0110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f7a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0110850 con 0x7f10f00fce40 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f7a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0110f90 con 0x7f10f00fe040 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 0x7f10f010fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 0x7f10f010fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49636/0 (socket says 172.21.15.44:49636) 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10f5000700 1 -- 172.21.15.44:0/1315636796 learned_addr learned my addr 172.21.15.44:0/1315636796 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.592+0000 7f10efe00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f00fe040 0x7f10f0110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10f5000700 1 -- 172.21.15.44:0/1315636796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f00fe040 msgr2=0x7f10f0110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:33.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10f5000700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f00fe040 0x7f10f0110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10f5000700 1 -- 172.21.15.44:0/1315636796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc005040 con 0x7f10f00fce40 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10f5000700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 0x7f10f010fd10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f10e400c8b0 tx=0x7f10e4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e400a760 con 0x7f10f00fce40 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10e400ad90 con 0x7f10f00fce40 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e40113a0 con 0x7f10f00fce40 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.593+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f0111270 con 0x7f10f00fce40 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.594+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f0111830 con 0x7f10f00fce40 2024-04-16T11:48:33.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.595+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f01019d0 con 0x7f10f00fce40 2024-04-16T11:48:33.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.596+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f10e4012400 con 0x7f10f00fce40 2024-04-16T11:48:33.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.597+0000 7f10ecc00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f10d806ec00 0x7f10d80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:33.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.597+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f10e408e960 con 0x7f10f00fce40 2024-04-16T11:48:33.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.597+0000 7f10efe00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f10d806ec00 0x7f10d80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:33.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.598+0000 7f10efe00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f10d806ec00 0x7f10d80710b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00b3b0 tx=0x7f10dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:33.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.603+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f10e405b1a0 con 0x7f10f00fce40 2024-04-16T11:48:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.853+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10f0111b00 con 0x7f10d806ec00 2024-04-16T11:48:33.850 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:33 smithi044 ceph-mon[93020]: pgmap v147: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:33.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.855+0000 7f10ecc00700 1 -- 172.21.15.44:0/1315636796 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f10f0111b00 con 0x7f10d806ec00 2024-04-16T11:48:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f10d806ec00 msgr2=0x7f10d80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f10d806ec00 0x7f10d80710b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00b3b0 tx=0x7f10dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 msgr2=0x7f10f010fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 0x7f10f010fd10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f10e400c8b0 tx=0x7f10e4004470 comp rx=0 tx=0).stop 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 shutdown_connections 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f10d806ec00 0x7f10d80710b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f10f00fce40 0x7f10f010fd10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 --2- 172.21.15.44:0/1315636796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f00fe040 0x7f10f0110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 >> 172.21.15.44:0/1315636796 conn(0x7f10f00f8530 msgr2=0x7f10f00f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 shutdown_connections 2024-04-16T11:48:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:33.858+0000 7f10f7a00700 1 -- 172.21.15.44:0/1315636796 wait complete. 2024-04-16T11:48:33.871 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:48:34.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.012+0000 7f9335000700 1 -- 172.21.15.44:0/2132755820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 msgr2=0x7f9330104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.012+0000 7f9335000700 1 --2- 172.21.15.44:0/2132755820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f9330104b40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f931c004880 tx=0x7f931c009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.012+0000 7f9335000700 1 -- 172.21.15.44:0/2132755820 shutdown_connections 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.012+0000 7f9335000700 1 --2- 172.21.15.44:0/2132755820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f9330104b40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.012+0000 7f9335000700 1 --2- 172.21.15.44:0/2132755820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.012+0000 7f9335000700 1 -- 172.21.15.44:0/2132755820 >> 172.21.15.44:0/2132755820 conn(0x7f9330072640 msgr2=0x7f9330074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.013+0000 7f9335000700 1 -- 172.21.15.44:0/2132755820 shutdown_connections 2024-04-16T11:48:34.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.013+0000 7f9335000700 1 -- 172.21.15.44:0/2132755820 wait complete. 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f9335000700 1 Processor -- start 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f9335000700 1 -- start start 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f9335000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f9335000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f93301144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f9335000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330114aa0 con 0x7f9330103420 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f9335000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330114c10 con 0x7f93301046d0 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f932de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f932de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49652/0 (socket says 172.21.15.44:49652) 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.014+0000 7f932de00700 1 -- 172.21.15.44:0/2197351759 learned_addr learned my addr 172.21.15.44:0/2197351759 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.015+0000 7f932d400700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f93301144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.015+0000 7f932de00700 1 -- 172.21.15.44:0/2197351759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 msgr2=0x7f93301144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.015+0000 7f932de00700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f93301144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.015+0000 7f932de00700 1 -- 172.21.15.44:0/2197351759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c005040 con 0x7f9330103420 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.015+0000 7f932d400700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f93301144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.015+0000 7f932de00700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330113f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f93200041d0 tx=0x7f932000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.016+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9320007cc0 con 0x7f9330103420 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.016+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f932000f040 con 0x7f9330103420 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.016+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f932000ca50 con 0x7f9330103420 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.016+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9330119610 con 0x7f9330103420 2024-04-16T11:48:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.016+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9330119b80 con 0x7f9330103420 2024-04-16T11:48:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.017+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933004c710 con 0x7f9330103420 2024-04-16T11:48:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.018+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f932000ccc0 con 0x7f9330103420 2024-04-16T11:48:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.019+0000 7f9326000700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f931406ec00 0x7f93140710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.019+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f932008e470 con 0x7f9330103420 2024-04-16T11:48:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.019+0000 7f932d400700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f931406ec00 0x7f93140710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.020+0000 7f932d400700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f931406ec00 0x7f93140710b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f931c009fd0 tx=0x7f931c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.027+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f932005ac60 con 0x7f9330103420 2024-04-16T11:48:34.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:33 smithi045 ceph-mon[81423]: pgmap v147: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.278+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9330108a30 con 0x7f931406ec00 2024-04-16T11:48:34.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.281+0000 7f9326000700 1 -- 172.21.15.44:0/2197351759 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f9330108a30 con 0x7f931406ec00 2024-04-16T11:48:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f931406ec00 msgr2=0x7f93140710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f931406ec00 0x7f93140710b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f931c009fd0 tx=0x7f931c009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 msgr2=0x7f9330113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330113f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f93200041d0 tx=0x7f932000b040 comp rx=0 tx=0).stop 2024-04-16T11:48:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 shutdown_connections 2024-04-16T11:48:34.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f931406ec00 0x7f93140710b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9330103420 0x7f9330113f60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 --2- 172.21.15.44:0/2197351759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93301046d0 0x7f93301144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.284+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 >> 172.21.15.44:0/2197351759 conn(0x7f9330072640 msgr2=0x7f9330073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.285+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 shutdown_connections 2024-04-16T11:48:34.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.285+0000 7f9335000700 1 -- 172.21.15.44:0/2197351759 wait complete. 2024-04-16T11:48:34.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.441+0000 7f7ad8c00700 1 -- 172.21.15.44:0/180474955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 msgr2=0x7f7ad4100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.441+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/180474955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4100740 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0004940 tx=0x7f7ac0009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.441+0000 7f7ad8c00700 1 -- 172.21.15.44:0/180474955 shutdown_connections 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.441+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/180474955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4100740 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.441+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/180474955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ad40ff020 0x7f7ad40ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.441+0000 7f7ad8c00700 1 -- 172.21.15.44:0/180474955 >> 172.21.15.44:0/180474955 conn(0x7f7ad40fa950 msgr2=0x7f7ad40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.442+0000 7f7ad8c00700 1 -- 172.21.15.44:0/180474955 shutdown_connections 2024-04-16T11:48:34.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.442+0000 7f7ad8c00700 1 -- 172.21.15.44:0/180474955 wait complete. 2024-04-16T11:48:34.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.442+0000 7f7ad8c00700 1 Processor -- start 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad8c00700 1 -- start start 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad8c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ad40ff020 0x7f7ad410fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad8c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad8c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad4110720 con 0x7f7ad41002d0 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad8c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad41150c0 con 0x7f7ad40ff020 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad1400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49670/0 (socket says 172.21.15.44:49670) 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad1400700 1 -- 172.21.15.44:0/2868344588 learned_addr learned my addr 172.21.15.44:0/2868344588 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.443+0000 7f7ad1e00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ad40ff020 0x7f7ad410fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7ad1400700 1 -- 172.21.15.44:0/2868344588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ad40ff020 msgr2=0x7f7ad410fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7ad1400700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ad40ff020 0x7f7ad410fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7ad1400700 1 -- 172.21.15.44:0/2868344588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0005040 con 0x7f7ad41002d0 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7ad1400700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4110090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000b3b0 tx=0x7f7ac0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ac000e040 con 0x7f7ad41002d0 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ac0004290 con 0x7f7ad41002d0 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ac001c630 con 0x7f7ad41002d0 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad41152c0 con 0x7f7ad41002d0 2024-04-16T11:48:34.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.444+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad4115830 con 0x7f7ad41002d0 2024-04-16T11:48:34.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.446+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad404c710 con 0x7f7ad41002d0 2024-04-16T11:48:34.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.447+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ac000bbf0 con 0x7f7ad41002d0 2024-04-16T11:48:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.448+0000 7f7aca000700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7ab806ec00 0x7f7ab80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.448+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f7ac008fda0 con 0x7f7ad41002d0 2024-04-16T11:48:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.448+0000 7f7ad1e00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7ab806ec00 0x7f7ab80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.449+0000 7f7ad1e00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7ab806ec00 0x7f7ab80710b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4006fd0 tx=0x7f7ac4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:34.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.453+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ac005c600 con 0x7f7ad41002d0 2024-04-16T11:48:34.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.682+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ad4104570 con 0x7f7ab806ec00 2024-04-16T11:48:34.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.713+0000 7f7aca000700 1 -- 172.21.15.44:0/2868344588 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f7ad4104570 con 0x7f7ab806ec00 2024-04-16T11:48:34.710 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:48:34.710 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (5m) 41s ago 16m 13.2M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (2m) 41s ago 16m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (2m) 13s ago 15m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (5m) 41s ago 15m 74.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (8m) 41s ago 8m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (9m) 41s ago 9m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (8m) 13s ago 8m 13.8M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (8m) 13s ago 8m 13.9M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (5m) 41s ago 17m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (4m) 13s ago 14m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (4m) 41s ago 18m 46.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:48:34.711 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (3m) 13s ago 14m 33.0M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (5m) 41s ago 16m 7880k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (5m) 13s ago 14m 8040k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (110s) 41s ago 13m 50.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (79s) 41s ago 13m 42.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (49s) 41s ago 12m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (18s) 13s ago 11m 11.2M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (11m) 13s ago 11m 47.5M 2583M 17.2.7-847-g45317f95 b333581ccdf5 ac83c9ff77e7 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (10m) 13s ago 10m 49.0M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:48:34.712 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (4m) 41s ago 15m 56.6M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:48:34.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7ab806ec00 msgr2=0x7f7ab80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7ab806ec00 0x7f7ab80710b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4006fd0 tx=0x7f7ac4008040 comp rx=0 tx=0).stop 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 msgr2=0x7f7ad4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4110090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000b3b0 tx=0x7f7ac0009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 shutdown_connections 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7ab806ec00 0x7f7ab80710b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ad40ff020 0x7f7ad410fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 --2- 172.21.15.44:0/2868344588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41002d0 0x7f7ad4110090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 >> 172.21.15.44:0/2868344588 conn(0x7f7ad40fa950 msgr2=0x7f7ad40fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.716+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 shutdown_connections 2024-04-16T11:48:34.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.717+0000 7f7ad8c00700 1 -- 172.21.15.44:0/2868344588 wait complete. 2024-04-16T11:48:34.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3946095450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc1002d0 msgr2=0x7f1fcc100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3946095450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc1002d0 0x7f1fcc100740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4004940 tx=0x7f1fb4009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:34.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3946095450 shutdown_connections 2024-04-16T11:48:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3946095450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc1002d0 0x7f1fcc100740 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3946095450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc0ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3946095450 >> 172.21.15.44:0/3946095450 conn(0x7f1fcc0fa950 msgr2=0x7f1fcc0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3946095450 shutdown_connections 2024-04-16T11:48:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.858+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3946095450 wait complete. 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.859+0000 7f1fd0a00700 1 Processor -- start 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.859+0000 7f1fd0a00700 1 -- start start 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fd0a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc10fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fd0a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fcc1002d0 0x7f1fcc110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fd0a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc110720 con 0x7f1fcc0ff020 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fd0a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc1150c0 con 0x7f1fcc1002d0 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc10fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc9e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc10fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49680/0 (socket says 172.21.15.44:49680) 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc9e00700 1 -- 172.21.15.44:0/3719141204 learned_addr learned my addr 172.21.15.44:0/3719141204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc3e00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fcc1002d0 0x7f1fcc110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc9e00700 1 -- 172.21.15.44:0/3719141204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fcc1002d0 msgr2=0x7f1fcc110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:34.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc9e00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fcc1002d0 0x7f1fcc110090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:34.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.860+0000 7f1fc9e00700 1 -- 172.21.15.44:0/3719141204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4005040 con 0x7f1fcc0ff020 2024-04-16T11:48:34.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.861+0000 7f1fc9e00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc10fb50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00b790 tx=0x7f1fbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:34.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.861+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fbc00f770 con 0x7f1fcc0ff020 2024-04-16T11:48:34.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.861+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fbc00fda0 con 0x7f1fcc0ff020 2024-04-16T11:48:34.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.861+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fbc00d310 con 0x7f1fcc0ff020 2024-04-16T11:48:34.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.861+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc115320 con 0x7f1fcc0ff020 2024-04-16T11:48:34.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.861+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc115890 con 0x7f1fcc0ff020 2024-04-16T11:48:34.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.862+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc04c710 con 0x7f1fcc0ff020 2024-04-16T11:48:34.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.863+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1fbc010420 con 0x7f1fcc0ff020 2024-04-16T11:48:34.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.864+0000 7f1fc2000700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1fb006ec00 0x7f1fb00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:34.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.864+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f1fbc08e990 con 0x7f1fcc0ff020 2024-04-16T11:48:34.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.865+0000 7f1fc3e00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1fb006ec00 0x7f1fb00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:34.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.865+0000 7f1fc3e00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1fb006ec00 0x7f1fb00710b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4004f90 tx=0x7f1fb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:34.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:34.869+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1fbc05b1d0 con 0x7f1fcc0ff020 2024-04-16T11:48:34.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:34 smithi044 ceph-mon[93020]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:35.000 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:34 smithi045 ceph-mon[81423]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:35.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.193+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1fcc115b60 con 0x7f1fcc0ff020 2024-04-16T11:48:35.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.194+0000 7f1fc2000700 1 -- 172.21.15.44:0/3719141204 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f1fbc05ad60 con 0x7f1fcc0ff020 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:35.191 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6, 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 8 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:48:35.192 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:48:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1fb006ec00 msgr2=0x7f1fb00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1fb006ec00 0x7f1fb00710b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4004f90 tx=0x7f1fb4009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:35.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 msgr2=0x7f1fcc10fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc10fb50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00b790 tx=0x7f1fbc009250 comp rx=0 tx=0).stop 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 shutdown_connections 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1fb006ec00 0x7f1fb00710b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1fcc0ff020 0x7f1fcc10fb50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 --2- 172.21.15.44:0/3719141204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fcc1002d0 0x7f1fcc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 >> 172.21.15.44:0/3719141204 conn(0x7f1fcc0fa950 msgr2=0x7f1fcc0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 shutdown_connections 2024-04-16T11:48:35.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.197+0000 7f1fd0a00700 1 -- 172.21.15.44:0/3719141204 wait complete. 2024-04-16T11:48:35.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 -- 172.21.15.44:0/2100372370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 msgr2=0x7fa5f40fbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 --2- 172.21.15.44:0/2100372370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f40fbf40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8004940 tx=0x7fa5e8009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:35.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 -- 172.21.15.44:0/2100372370 shutdown_connections 2024-04-16T11:48:35.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 --2- 172.21.15.44:0/2100372370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5f40fcde0 0x7fa5f40fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 --2- 172.21.15.44:0/2100372370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f40fbf40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 -- 172.21.15.44:0/2100372370 >> 172.21.15.44:0/2100372370 conn(0x7fa5f40f7290 msgr2=0x7fa5f40f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:35.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 -- 172.21.15.44:0/2100372370 shutdown_connections 2024-04-16T11:48:35.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.356+0000 7fa5fb000700 1 -- 172.21.15.44:0/2100372370 wait complete. 2024-04-16T11:48:35.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.357+0000 7fa5fb000700 1 Processor -- start 2024-04-16T11:48:35.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.357+0000 7fa5fb000700 1 -- start start 2024-04-16T11:48:35.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5fb000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f410fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5fb000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5f40fcde0 0x7fa5f410fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5fb000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f41105f0 con 0x7fa5f40fbb30 2024-04-16T11:48:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5fb000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4110df0 con 0x7fa5f40fcde0 2024-04-16T11:48:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f410fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f410fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49702/0 (socket says 172.21.15.44:49702) 2024-04-16T11:48:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9c00700 1 -- 172.21.15.44:0/1781903791 learned_addr learned my addr 172.21.15.44:0/1781903791 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9200700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5f40fcde0 0x7fa5f410fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9c00700 1 -- 172.21.15.44:0/1781903791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5f40fcde0 msgr2=0x7fa5f410fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9c00700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5f40fcde0 0x7fa5f410fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.358+0000 7fa5f9c00700 1 -- 172.21.15.44:0/1781903791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8005040 con 0x7fa5f40fbb30 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.359+0000 7fa5f9c00700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f410fab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009150 tx=0x7fa5e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.359+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5e800e040 con 0x7fa5f40fbb30 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.359+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5e800bbf0 con 0x7fa5f40fbb30 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.359+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f4111070 con 0x7fa5f40fbb30 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.359+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5e801c630 con 0x7fa5f40fbb30 2024-04-16T11:48:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.359+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f41115b0 con 0x7fa5f40fbb30 2024-04-16T11:48:35.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.360+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f4101770 con 0x7fa5f40fbb30 2024-04-16T11:48:35.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.361+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa5e800bd60 con 0x7fa5f40fbb30 2024-04-16T11:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.362+0000 7fa5f1600700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5dc06ec10 0x7fa5dc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.362+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fa5e808fd30 con 0x7fa5f40fbb30 2024-04-16T11:48:35.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.363+0000 7fa5f9200700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5dc06ec10 0x7fa5dc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:35.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.363+0000 7fa5f9200700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5dc06ec10 0x7fa5dc0710c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0004d60 tx=0x7fa5e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:35.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.368+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa5e805c590 con 0x7fa5f40fbb30 2024-04-16T11:48:35.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.644+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa5f4111890 con 0x7fa5f40fbb30 2024-04-16T11:48:35.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.645+0000 7fa5f1600700 1 -- 172.21.15.44:0/1781903791 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7fa5e805c120 con 0x7fa5f40fbb30 2024-04-16T11:48:35.643 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:48:35.643 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:48:35.643 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:48:35.643 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:48:35.644 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:35.645 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.649+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5dc06ec10 msgr2=0x7fa5dc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.649+0000 7fa5fb000700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5dc06ec10 0x7fa5dc0710c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0004d60 tx=0x7fa5e000b040 comp rx=0 tx=0).stop 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.649+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 msgr2=0x7fa5f410fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.649+0000 7fa5fb000700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f410fab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009150 tx=0x7fa5e8009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 shutdown_connections 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5dc06ec10 0x7fa5dc0710c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa5f40fbb30 0x7fa5f410fab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 --2- 172.21.15.44:0/1781903791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5f40fcde0 0x7fa5f410fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 >> 172.21.15.44:0/1781903791 conn(0x7fa5f40f7290 msgr2=0x7fa5f4066430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 shutdown_connections 2024-04-16T11:48:35.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.650+0000 7fa5fb000700 1 -- 172.21.15.44:0/1781903791 wait complete. 2024-04-16T11:48:35.648 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:48:35.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 -- 172.21.15.44:0/758985633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 msgr2=0x7f2a300ffdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 --2- 172.21.15.44:0/758985633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 0x7f2a300ffdb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2a24004880 tx=0x7f2a24009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:35.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 -- 172.21.15.44:0/758985633 shutdown_connections 2024-04-16T11:48:35.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 --2- 172.21.15.44:0/758985633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a301026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 --2- 172.21.15.44:0/758985633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 0x7f2a300ffdb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 -- 172.21.15.44:0/758985633 >> 172.21.15.44:0/758985633 conn(0x7f2a300f7450 msgr2=0x7f2a300f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:35.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 -- 172.21.15.44:0/758985633 shutdown_connections 2024-04-16T11:48:35.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.806+0000 7f2a38800700 1 -- 172.21.15.44:0/758985633 wait complete. 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.807+0000 7f2a38800700 1 Processor -- start 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.807+0000 7f2a38800700 1 -- start start 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a38800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 0x7f2a300fd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a38800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a300fb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a38800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300fbe90 con 0x7f2a301002f0 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a38800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300fc000 con 0x7f2a300fd9d0 2024-04-16T11:48:35.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a35400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a300fb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:35.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a35400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a300fb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49718/0 (socket says 172.21.15.44:49718) 2024-04-16T11:48:35.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a35400700 1 -- 172.21.15.44:0/2642981811 learned_addr learned my addr 172.21.15.44:0/2642981811 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:35.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.808+0000 7f2a35e00700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 0x7f2a300fd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:35.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a35400700 1 -- 172.21.15.44:0/2642981811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 msgr2=0x7f2a300fd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:35.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a35400700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 0x7f2a300fd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:35.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a35400700 1 -- 172.21.15.44:0/2642981811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24005040 con 0x7f2a301002f0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a35400700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a300fb8e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a200041d0 tx=0x7f2a2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a20007cc0 con 0x7f2a301002f0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a2000f040 con 0x7f2a301002f0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a300fc280 con 0x7f2a301002f0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2000ca50 con 0x7f2a301002f0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.809+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a30115a80 con 0x7f2a301002f0 2024-04-16T11:48:35.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.810+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3005f5f0 con 0x7f2a301002f0 2024-04-16T11:48:35.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.811+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2a2000cbb0 con 0x7f2a301002f0 2024-04-16T11:48:35.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.812+0000 7f2a2a000700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2a1806ec00 0x7f2a180710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:35.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.812+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f2a2008e390 con 0x7f2a301002f0 2024-04-16T11:48:35.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.813+0000 7f2a35e00700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2a1806ec00 0x7f2a180710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:35.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.813+0000 7f2a35e00700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2a1806ec00 0x7f2a180710b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2a24000c00 tx=0x7f2a24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:35.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:35.817+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2a200570d0 con 0x7f2a301002f0 2024-04-16T11:48:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:35 smithi044 ceph-mon[93020]: pgmap v148: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:35 smithi044 ceph-mon[93020]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:35 smithi044 ceph-mon[93020]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:35 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3719141204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:35 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1781903791' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:48:36.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.061+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a300f8f90 con 0x7f2a1806ec00 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.063+0000 7f2a2a000700 1 -- 172.21.15.44:0/2642981811 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2a300f8f90 con 0x7f2a1806ec00 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "10/21 daemons upgraded", 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T11:48:36.060 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:48:36.061 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:48:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2a1806ec00 msgr2=0x7f2a180710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2a1806ec00 0x7f2a180710b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2a24000c00 tx=0x7f2a24009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 msgr2=0x7f2a300fb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a300fb8e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a200041d0 tx=0x7f2a2000b040 comp rx=0 tx=0).stop 2024-04-16T11:48:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 shutdown_connections 2024-04-16T11:48:36.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f2a1806ec00 0x7f2a180710b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a300fd9d0 0x7f2a300fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 --2- 172.21.15.44:0/2642981811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a301002f0 0x7f2a300fb8e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 >> 172.21.15.44:0/2642981811 conn(0x7f2a300f7450 msgr2=0x7f2a300f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:36.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.066+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 shutdown_connections 2024-04-16T11:48:36.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.067+0000 7f2a38800700 1 -- 172.21.15.44:0/2642981811 wait complete. 2024-04-16T11:48:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:35 smithi045 ceph-mon[81423]: pgmap v148: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:35 smithi045 ceph-mon[81423]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:35 smithi045 ceph-mon[81423]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:35 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3719141204' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:35 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1781903791' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 -- 172.21.15.44:0/3582727686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 msgr2=0x7fa664100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3582727686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa664100790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa650004940 tx=0x7fa650009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 -- 172.21.15.44:0/3582727686 shutdown_connections 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3582727686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa664100790 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3582727686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6640ff070 0x7fa6640ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 -- 172.21.15.44:0/3582727686 >> 172.21.15.44:0/3582727686 conn(0x7fa6640fa950 msgr2=0x7fa6640fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 -- 172.21.15.44:0/3582727686 shutdown_connections 2024-04-16T11:48:36.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.218+0000 7fa66bc00700 1 -- 172.21.15.44:0/3582727686 wait complete. 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.219+0000 7fa66bc00700 1 Processor -- start 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.219+0000 7fa66bc00700 1 -- start start 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.219+0000 7fa66bc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6640ff070 0x7fa66410fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.219+0000 7fa66bc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa6641100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.219+0000 7fa66bc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6641106e0 con 0x7fa664100320 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.219+0000 7fa66bc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa664110f30 con 0x7fa6640ff070 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa6641100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa6641100e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49730/0 (socket says 172.21.15.44:49730) 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 -- 172.21.15.44:0/3161639809 learned_addr learned my addr 172.21.15.44:0/3161639809 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa669200700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6640ff070 0x7fa66410fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 -- 172.21.15.44:0/3161639809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6640ff070 msgr2=0x7fa66410fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:36.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6640ff070 0x7fa66410fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 -- 172.21.15.44:0/3161639809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650005040 con 0x7fa664100320 2024-04-16T11:48:36.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.220+0000 7fa663e00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa6641100e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa650013870 tx=0x7fa650009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:36.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.221+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa65000e070 con 0x7fa664100320 2024-04-16T11:48:36.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.221+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6641111b0 con 0x7fa664100320 2024-04-16T11:48:36.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.221+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa65001c400 con 0x7fa664100320 2024-04-16T11:48:36.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.221+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa664111690 con 0x7fa664100320 2024-04-16T11:48:36.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.221+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa650026640 con 0x7fa664100320 2024-04-16T11:48:36.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.222+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66405f5f0 con 0x7fa664100320 2024-04-16T11:48:36.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.223+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa65001c570 con 0x7fa664100320 2024-04-16T11:48:36.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.224+0000 7fa660c00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa64c06ec10 0x7fa64c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:48:36.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.224+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fa650090150 con 0x7fa664100320 2024-04-16T11:48:36.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.225+0000 7fa669200700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa64c06ec10 0x7fa64c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:48:36.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.225+0000 7fa669200700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa64c06ec10 0x7fa64c0710c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa658004600 tx=0x7fa65800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:48:36.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.229+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa65005c910 con 0x7fa664100320 2024-04-16T11:48:36.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.553+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa66404bed0 con 0x7fa664100320 2024-04-16T11:48:36.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.554+0000 7fa660c00700 1 -- 172.21.15.44:0/3161639809 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fa65005c4a0 con 0x7fa664100320 2024-04-16T11:48:36.551 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.556+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa64c06ec10 msgr2=0x7fa64c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.556+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa64c06ec10 0x7fa64c0710c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa658004600 tx=0x7fa65800b040 comp rx=0 tx=0).stop 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.556+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 msgr2=0x7fa6641100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.556+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa6641100e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa650013870 tx=0x7fa650009e30 comp rx=0 tx=0).stop 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 shutdown_connections 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa64c06ec10 0x7fa64c0710c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6640ff070 0x7fa66410fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 --2- 172.21.15.44:0/3161639809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa664100320 0x7fa6641100e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:48:36.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 >> 172.21.15.44:0/3161639809 conn(0x7fa6640fa950 msgr2=0x7fa6640fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:48:36.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 shutdown_connections 2024-04-16T11:48:36.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:48:36.557+0000 7fa66bc00700 1 -- 172.21.15.44:0/3161639809 wait complete. 2024-04-16T11:48:36.579 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:36 smithi045 systemd[1]: Stopping Ceph osd.4 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:48:36.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:36 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3161639809' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:48:37.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:36 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3161639809' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:48:37.079 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:36 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[48372]: 2024-04-16T11:48:36.751+0000 7ffbbb600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:48:37.079 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:36 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[48372]: 2024-04-16T11:48:36.751+0000 7ffbbb600700 -1 osd.4 76 *** Got signal Terminated *** 2024-04-16T11:48:37.080 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:36 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[48372]: 2024-04-16T11:48:36.751+0000 7ffbbb600700 -1 osd.4 76 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T11:48:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:37 smithi045 ceph-mon[81423]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:37 smithi045 ceph-mon[81423]: pgmap v149: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:37 smithi045 ceph-mon[81423]: osd.4 marked itself down and dead 2024-04-16T11:48:38.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:37 smithi044 ceph-mon[93020]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:48:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:37 smithi044 ceph-mon[93020]: pgmap v149: 65 pgs: 65 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:37 smithi044 ceph-mon[93020]: osd.4 marked itself down and dead 2024-04-16T11:48:38.631 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:38 smithi045 podman[95572]: 2024-04-16 11:48:38.340871321 +0000 UTC m=+1.713749655 container died ac83c9ff77e7a9bf279e2439a816f9747f355bcd014c5f076bf55183760a7178 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-16T11:48:38.952 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:38 smithi045 ceph-mon[81423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:48:38.952 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:38 smithi045 ceph-mon[81423]: osdmap e77: 6 total, 5 up, 6 in 2024-04-16T11:48:38.952 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:38 smithi045 podman[95572]: 2024-04-16 11:48:38.761947686 +0000 UTC m=+2.134826050 container cleanup ac83c9ff77e7a9bf279e2439a816f9747f355bcd014c5f076bf55183760a7178 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, ceph=True, CEPH_POINT_RELEASE=) 2024-04-16T11:48:38.952 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:38 smithi045 bash[95572]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4 2024-04-16T11:48:39.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:38 smithi044 ceph-mon[93020]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:48:39.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:38 smithi044 ceph-mon[93020]: osdmap e77: 6 total, 5 up, 6 in 2024-04-16T11:48:39.329 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:38 smithi045 podman[95591]: 2024-04-16 11:48:38.954849411 +0000 UTC m=+0.599431012 container remove ac83c9ff77e7a9bf279e2439a816f9747f355bcd014c5f076bf55183760a7178 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T11:48:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:39 smithi045 ceph-mon[81423]: pgmap v151: 65 pgs: 6 stale+active+clean, 59 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:39 smithi045 ceph-mon[81423]: osdmap e78: 6 total, 5 up, 6 in 2024-04-16T11:48:40.080 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:39 smithi045 podman[95683]: 2024-04-16 11:48:39.763964833 +0000 UTC m=+0.037672338 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:48:40.080 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:39 smithi045 podman[95683]: 2024-04-16 11:48:39.897430619 +0000 UTC m=+0.171138142 container create bfcf268ed1295c76c44510776dc5c5d21e93969287860600f69127e8504cd0f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T11:48:40.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:39 smithi044 ceph-mon[93020]: pgmap v151: 65 pgs: 6 stale+active+clean, 59 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:40.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:39 smithi044 ceph-mon[93020]: osdmap e78: 6 total, 5 up, 6 in 2024-04-16T11:48:40.579 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:40 smithi045 podman[95683]: 2024-04-16 11:48:40.21799832 +0000 UTC m=+0.491705866 container init bfcf268ed1295c76c44510776dc5c5d21e93969287860600f69127e8504cd0f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-16T11:48:40.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:40 smithi045 podman[95683]: 2024-04-16 11:48:40.231465543 +0000 UTC m=+0.505173093 container start bfcf268ed1295c76c44510776dc5c5d21e93969287860600f69127e8504cd0f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T11:48:40.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:40 smithi045 podman[95683]: 2024-04-16 11:48:40.333195129 +0000 UTC m=+0.606902641 container attach bfcf268ed1295c76c44510776dc5c5d21e93969287860600f69127e8504cd0f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T11:48:41.038 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:40 smithi045 podman[95683]: 2024-04-16 11:48:40.689463131 +0000 UTC m=+0.963170677 container died bfcf268ed1295c76c44510776dc5c5d21e93969287860600f69127e8504cd0f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T11:48:41.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:41 smithi045 ceph-mon[81423]: pgmap v153: 65 pgs: 9 peering, 4 stale+active+clean, 52 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:41 smithi045 ceph-mon[81423]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T11:48:41.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:41 smithi044 ceph-mon[93020]: pgmap v153: 65 pgs: 9 peering, 4 stale+active+clean, 52 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:41.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:41.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:41.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:41 smithi044 ceph-mon[93020]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T11:48:41.829 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:41 smithi045 podman[95708]: 2024-04-16 11:48:41.437111637 +0000 UTC m=+0.729829555 container remove bfcf268ed1295c76c44510776dc5c5d21e93969287860600f69127e8504cd0f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T11:48:41.830 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:41 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-16T11:48:41.830 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:41 smithi045 systemd[1]: Stopped Ceph osd.4 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:48:41.830 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:41 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.4.service: Consumed 13.306s CPU time. 2024-04-16T11:48:42.277 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:41 smithi045 systemd[1]: Starting Ceph osd.4 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:48:42.548 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:42 smithi045 ceph-mon[81423]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:42.548 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:42 smithi045 podman[95805]: 2024-04-16 11:48:42.280190968 +0000 UTC m=+0.120734735 container create 9498803e8bf640cd2aa55ce6707b99833d618483926fe47a57ac43a50bc82cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:48:42.549 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:42 smithi045 podman[95805]: 2024-04-16 11:48:42.19485776 +0000 UTC m=+0.035401489 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:48:42.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:42 smithi044 ceph-mon[93020]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:42.830 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:42 smithi045 podman[95805]: 2024-04-16 11:48:42.551237296 +0000 UTC m=+0.391781069 container init 9498803e8bf640cd2aa55ce6707b99833d618483926fe47a57ac43a50bc82cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-16T11:48:42.830 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:42 smithi045 podman[95805]: 2024-04-16 11:48:42.563950632 +0000 UTC m=+0.404494403 container start 9498803e8bf640cd2aa55ce6707b99833d618483926fe47a57ac43a50bc82cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-16T11:48:42.830 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:42 smithi045 podman[95805]: 2024-04-16 11:48:42.674444795 +0000 UTC m=+0.514988608 container attach 9498803e8bf640cd2aa55ce6707b99833d618483926fe47a57ac43a50bc82cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-16T11:48:43.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 406, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 95927 2024-04-16T11:48:43.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 406, success, outsize: 120 2024-04-16T11:48:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-mon[81423]: pgmap v154: 65 pgs: 5 active+undersized+degraded, 9 peering, 18 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 bash[95805]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T11:48:43.580 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T11:48:43.581 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate[95820]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T11:48:43.581 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:43 smithi045 podman[95805]: 2024-04-16 11:48:43.491906376 +0000 UTC m=+1.332450137 container died 9498803e8bf640cd2aa55ce6707b99833d618483926fe47a57ac43a50bc82cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:48:43.627 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:43 smithi044 ceph-mon[93020]: pgmap v154: 65 pgs: 5 active+undersized+degraded, 9 peering, 18 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:48:44.545 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:44 smithi045 ceph-mon[81423]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T11:48:44.545 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 podman[95955]: 2024-04-16 11:48:44.16992465 +0000 UTC m=+0.657568822 container remove 9498803e8bf640cd2aa55ce6707b99833d618483926fe47a57ac43a50bc82cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5) 2024-04-16T11:48:44.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:44 smithi044 ceph-mon[93020]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T11:48:44.829 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 podman[96014]: 2024-04-16 11:48:44.455122152 +0000 UTC m=+0.039647796 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:48:44.829 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 podman[96014]: 2024-04-16 11:48:44.579156608 +0000 UTC m=+0.163682285 container create c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-16T11:48:45.329 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 podman[96014]: 2024-04-16 11:48:44.874237703 +0000 UTC m=+0.458763400 container init c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, ceph=True) 2024-04-16T11:48:45.329 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 podman[96014]: 2024-04-16 11:48:44.885862398 +0000 UTC m=+0.470388051 container start c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T11:48:45.330 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 ceph-osd[96032]: -- 172.21.15.45:0/376408773 <== mon.1 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562758329680 con 0x5627582f3000 2024-04-16T11:48:45.330 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 bash[96014]: c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a 2024-04-16T11:48:45.330 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:44 smithi045 systemd[1]: Started Ceph osd.4 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:48:45.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:45 smithi044 ceph-mon[93020]: pgmap v155: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:45 smithi045 ceph-mon[81423]: pgmap v155: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:45.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:45.829 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:45 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T11:48:45.471+0000 7f2d3e0c77c0 -1 Falling back to public interface 2024-04-16T11:48:47.678 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:47 smithi045 ceph-mon[81423]: pgmap v156: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:47.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:47 smithi044 ceph-mon[93020]: pgmap v156: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:49.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:49 smithi044 ceph-mon[93020]: pgmap v157: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:49.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:49 smithi045 ceph-mon[81423]: pgmap v157: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:49.829 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T11:48:49.399+0000 7f2d3e0c77c0 -1 osd.4 76 log_to_monitors true 2024-04-16T11:48:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:50 smithi044 ceph-mon[93020]: from='osd.4 [v2:172.21.15.45:6808/3529709154,v1:172.21.15.45:6809/3529709154]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:48:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:50 smithi044 ceph-mon[93020]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:48:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:50.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:50 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:50.829 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 11:48:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T11:48:50.418+0000 7f2d2d400700 -1 osd.4 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:48:50.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:50 smithi045 ceph-mon[81423]: from='osd.4 [v2:172.21.15.45:6808/3529709154,v1:172.21.15.45:6809/3529709154]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:48:50.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:50 smithi045 ceph-mon[81423]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T11:48:50.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:50.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:50.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:50 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:51 smithi044 ceph-mon[93020]: pgmap v158: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:51 smithi044 ceph-mon[93020]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T11:48:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:51 smithi044 ceph-mon[93020]: osdmap e79: 6 total, 5 up, 6 in 2024-04-16T11:48:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:51 smithi044 ceph-mon[93020]: from='osd.4 [v2:172.21.15.45:6808/3529709154,v1:172.21.15.45:6809/3529709154]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:51.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:51 smithi044 ceph-mon[93020]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:51 smithi045 ceph-mon[81423]: pgmap v158: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:51 smithi045 ceph-mon[81423]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T11:48:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:51 smithi045 ceph-mon[81423]: osdmap e79: 6 total, 5 up, 6 in 2024-04-16T11:48:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:51 smithi045 ceph-mon[81423]: from='osd.4 [v2:172.21.15.45:6808/3529709154,v1:172.21.15.45:6809/3529709154]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:51.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:51 smithi045 ceph-mon[81423]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: osd.4 [v2:172.21.15.45:6808/3529709154,v1:172.21.15.45:6809/3529709154] boot 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: osdmap e80: 6 total, 6 up, 6 in 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:52.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:52 smithi044 ceph-mon[93020]: osdmap e81: 6 total, 6 up, 6 in 2024-04-16T11:48:52.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:48:52.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: osd.4 [v2:172.21.15.45:6808/3529709154,v1:172.21.15.45:6809/3529709154] boot 2024-04-16T11:48:52.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: osdmap e80: 6 total, 6 up, 6 in 2024-04-16T11:48:52.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T11:48:52.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:52.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:52.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:52 smithi045 ceph-mon[81423]: osdmap e81: 6 total, 6 up, 6 in 2024-04-16T11:48:54.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:53 smithi045 ceph-mon[81423]: pgmap v161: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 439 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:54.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:53 smithi044 ceph-mon[93020]: pgmap v161: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 439 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T11:48:55.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 408, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97756 2024-04-16T11:48:55.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 408, success, outsize: 120 2024-04-16T11:48:55.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 410, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97850 2024-04-16T11:48:55.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 410, success, outsize: 120 2024-04-16T11:48:55.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 412, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97947 2024-04-16T11:48:55.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 412, success, outsize: 120 2024-04-16T11:48:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 414, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98050 2024-04-16T11:48:56.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 414, success, outsize: 120 2024-04-16T11:48:56.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 416, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98150 2024-04-16T11:48:56.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 416, success, outsize: 120 2024-04-16T11:48:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:55 smithi045 ceph-mon[81423]: pgmap v163: 65 pgs: 6 active+undersized+degraded, 19 active+undersized, 40 active+clean; 584 KiB data, 439 MiB used, 536 GiB / 536 GiB avail; 9/72 objects degraded (12.500%) 2024-04-16T11:48:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:55 smithi045 ceph-mon[81423]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:56.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:56.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 418, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98244 2024-04-16T11:48:56.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 418, success, outsize: 120 2024-04-16T11:48:56.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 420, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98341 2024-04-16T11:48:56.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 420, success, outsize: 120 2024-04-16T11:48:56.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 422, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98444 2024-04-16T11:48:56.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 422, success, outsize: 120 2024-04-16T11:48:56.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 424, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98544 2024-04-16T11:48:56.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 424, success, outsize: 120 2024-04-16T11:48:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 426, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98638 2024-04-16T11:48:56.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 426, success, outsize: 120 2024-04-16T11:48:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:55 smithi044 ceph-mon[93020]: pgmap v163: 65 pgs: 6 active+undersized+degraded, 19 active+undersized, 40 active+clean; 584 KiB data, 439 MiB used, 536 GiB / 536 GiB avail; 9/72 objects degraded (12.500%) 2024-04-16T11:48:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:55 smithi044 ceph-mon[93020]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T11:48:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:48:56.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 428, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98735 2024-04-16T11:48:56.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 428, success, outsize: 120 2024-04-16T11:48:56.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 430, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98838 2024-04-16T11:48:56.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 430, success, outsize: 120 2024-04-16T11:48:56.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 432, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98938 2024-04-16T11:48:56.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 432, success, outsize: 120 2024-04-16T11:48:56.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 434, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99032 2024-04-16T11:48:56.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 434, success, outsize: 120 2024-04-16T11:48:56.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 436, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99129 2024-04-16T11:48:56.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 436, success, outsize: 120 2024-04-16T11:48:56.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99232 2024-04-16T11:48:56.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 438, success, outsize: 120 2024-04-16T11:48:56.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 440, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99332 2024-04-16T11:48:56.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 440, success, outsize: 120 2024-04-16T11:48:56.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 442, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99426 2024-04-16T11:48:56.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 442, success, outsize: 120 2024-04-16T11:48:56.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 444, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99523 2024-04-16T11:48:56.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 444, success, outsize: 120 2024-04-16T11:48:56.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 446, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99626 2024-04-16T11:48:56.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 446, success, outsize: 120 2024-04-16T11:48:56.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 448, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99726 2024-04-16T11:48:56.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 448, success, outsize: 120 2024-04-16T11:48:56.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 450, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99820 2024-04-16T11:48:56.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 450, success, outsize: 120 2024-04-16T11:48:56.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 452, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99917 2024-04-16T11:48:56.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 452, success, outsize: 120 2024-04-16T11:48:56.606 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 454, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100020 2024-04-16T11:48:56.606 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 454, success, outsize: 120 2024-04-16T11:48:56.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 456, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100120 2024-04-16T11:48:56.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 456, success, outsize: 120 2024-04-16T11:48:56.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 458, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100214 2024-04-16T11:48:56.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 458, success, outsize: 120 2024-04-16T11:48:56.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 460, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100311 2024-04-16T11:48:56.690 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 460, success, outsize: 120 2024-04-16T11:48:56.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 462, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100414 2024-04-16T11:48:56.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 462, success, outsize: 120 2024-04-16T11:48:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:56 smithi045 ceph-mon[81423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded) 2024-04-16T11:48:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:56 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:48:57.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:56 smithi044 ceph-mon[93020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded) 2024-04-16T11:48:57.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:56 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:48:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:57 smithi044 ceph-mon[93020]: pgmap v164: 65 pgs: 2 active+undersized, 63 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:48:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:57 smithi045 ceph-mon[81423]: pgmap v164: 65 pgs: 2 active+undersized, 63 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:48:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:48:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:48:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:48:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:58 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:58 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:58 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:58 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:58 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:48:59.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:58 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:48:59.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:58 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:59.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:58 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:59.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:58 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:48:59.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:58 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:59 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:59 smithi044 ceph-mon[93020]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-16T11:49:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:48:59 smithi044 ceph-mon[93020]: pgmap v165: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:59 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:59 smithi045 ceph-mon[81423]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-16T11:49:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:48:59 smithi045 ceph-mon[81423]: pgmap v165: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:01 smithi044 ceph-mon[93020]: pgmap v166: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:01 smithi045 ceph-mon[81423]: pgmap v166: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:04.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:03 smithi045 ceph-mon[81423]: pgmap v167: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:04.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:03 smithi044 ceph-mon[93020]: pgmap v167: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:06.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:05 smithi045 ceph-mon[81423]: pgmap v168: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:06.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:06.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:05 smithi044 ceph-mon[93020]: pgmap v168: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:06.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.724+0000 7f87d1c00700 1 -- 172.21.15.44:0/2319366869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 msgr2=0x7f87cc0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.724+0000 7f87d1c00700 1 --2- 172.21.15.44:0/2319366869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc0703b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f87bc004940 tx=0x7f87bc009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.724+0000 7f87d1c00700 1 -- 172.21.15.44:0/2319366869 shutdown_connections 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.724+0000 7f87d1c00700 1 --2- 172.21.15.44:0/2319366869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0708f0 0x7f87cc070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.724+0000 7f87d1c00700 1 --2- 172.21.15.44:0/2319366869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc0703b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.724+0000 7f87d1c00700 1 -- 172.21.15.44:0/2319366869 >> 172.21.15.44:0/2319366869 conn(0x7f87cc0f8540 msgr2=0x7f87cc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.725+0000 7f87d1c00700 1 -- 172.21.15.44:0/2319366869 shutdown_connections 2024-04-16T11:49:06.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.725+0000 7f87d1c00700 1 -- 172.21.15.44:0/2319366869 wait complete. 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.725+0000 7f87d1c00700 1 Processor -- start 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87d1c00700 1 -- start start 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87d1c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0708f0 0x7f87cc10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87d1c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87d1c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc1106b0 con 0x7f87cc071ea0 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87d1c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc110820 con 0x7f87cc0708f0 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87ca000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87ca000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc1100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36076/0 (socket says 172.21.15.44:36076) 2024-04-16T11:49:06.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.726+0000 7f87ca000700 1 -- 172.21.15.44:0/3925642586 learned_addr learned my addr 172.21.15.44:0/3925642586 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87caa00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0708f0 0x7f87cc10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87ca000700 1 -- 172.21.15.44:0/3925642586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0708f0 msgr2=0x7f87cc10fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:06.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87ca000700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0708f0 0x7f87cc10fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87ca000700 1 -- 172.21.15.44:0/3925642586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc005040 con 0x7f87cc071ea0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87ca000700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc1100b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f87b8002a50 tx=0x7f87b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b8007cc0 con 0x7f87cc071ea0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87b800f040 con 0x7f87cc071ea0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b8009640 con 0x7f87cc071ea0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.727+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc115280 con 0x7f87cc071ea0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.728+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc115840 con 0x7f87cc071ea0 2024-04-16T11:49:06.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.729+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc04c710 con 0x7f87cc071ea0 2024-04-16T11:49:06.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.730+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87b80097a0 con 0x7f87cc071ea0 2024-04-16T11:49:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.731+0000 7f87c2a00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87b006ec00 0x7f87b00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.731+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f87b808e6b0 con 0x7f87cc071ea0 2024-04-16T11:49:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.731+0000 7f87caa00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87b006ec00 0x7f87b00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:06.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.732+0000 7f87caa00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87b006ec00 0x7f87b00710b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00b3b0 tx=0x7f87bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:06.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.738+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f87b805af10 con 0x7f87cc071ea0 2024-04-16T11:49:06.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.990+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87cc103e70 con 0x7f87b006ec00 2024-04-16T11:49:06.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.992+0000 7f87c2a00700 1 -- 172.21.15.44:0/3925642586 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f87cc103e70 con 0x7f87b006ec00 2024-04-16T11:49:06.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87b006ec00 msgr2=0x7f87b00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:06.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87b006ec00 0x7f87b00710b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00b3b0 tx=0x7f87bc009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:06.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 msgr2=0x7f87cc1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:06.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc1100b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f87b8002a50 tx=0x7f87b80046b0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 shutdown_connections 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87b006ec00 0x7f87b00710b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87cc0708f0 0x7f87cc10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 --2- 172.21.15.44:0/3925642586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87cc071ea0 0x7f87cc1100b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.995+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 >> 172.21.15.44:0/3925642586 conn(0x7f87cc0f8540 msgr2=0x7f87cc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.996+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 shutdown_connections 2024-04-16T11:49:06.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:06.996+0000 7f87d1c00700 1 -- 172.21.15.44:0/3925642586 wait complete. 2024-04-16T11:49:07.010 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.156+0000 7f3400400700 1 -- 172.21.15.44:0/2156058478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f81046d0 msgr2=0x7f33f8104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.156+0000 7f3400400700 1 --2- 172.21.15.44:0/2156058478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f81046d0 0x7f33f8104b40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33e4004940 tx=0x7f33e4009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.157+0000 7f3400400700 1 -- 172.21.15.44:0/2156058478 shutdown_connections 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.157+0000 7f3400400700 1 --2- 172.21.15.44:0/2156058478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f81046d0 0x7f33f8104b40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.157+0000 7f3400400700 1 --2- 172.21.15.44:0/2156058478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33f8103420 0x7f33f8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.157+0000 7f3400400700 1 -- 172.21.15.44:0/2156058478 >> 172.21.15.44:0/2156058478 conn(0x7f33f8072640 msgr2=0x7f33f8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.157+0000 7f3400400700 1 -- 172.21.15.44:0/2156058478 shutdown_connections 2024-04-16T11:49:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.157+0000 7f3400400700 1 -- 172.21.15.44:0/2156058478 wait complete. 2024-04-16T11:49:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f3400400700 1 Processor -- start 2024-04-16T11:49:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f3400400700 1 -- start start 2024-04-16T11:49:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f3400400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 0x7f33f8113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f3400400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33f81046d0 0x7f33f8114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f3400400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8114a90 con 0x7f33f8103420 2024-04-16T11:49:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f3400400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8114c00 con 0x7f33f81046d0 2024-04-16T11:49:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f33fda00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 0x7f33f8113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f33fda00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 0x7f33f8113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36090/0 (socket says 172.21.15.44:36090) 2024-04-16T11:49:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.158+0000 7f33fda00700 1 -- 172.21.15.44:0/1446774755 learned_addr learned my addr 172.21.15.44:0/1446774755 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.159+0000 7f33fd000700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33f81046d0 0x7f33f8114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.159+0000 7f33fda00700 1 -- 172.21.15.44:0/1446774755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33f81046d0 msgr2=0x7f33f8114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.159+0000 7f33fda00700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33f81046d0 0x7f33f8114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.159+0000 7f33fda00700 1 -- 172.21.15.44:0/1446774755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4005040 con 0x7f33f8103420 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.159+0000 7f33fda00700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 0x7f33f8113f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c8c0 tx=0x7f33ec0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.160+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33ec00a980 con 0x7f33f8103420 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.160+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33ec00ce60 con 0x7f33f8103420 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.160+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33ec0115e0 con 0x7f33f8103420 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.160+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f81195f0 con 0x7f33f8103420 2024-04-16T11:49:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.160+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f8119bb0 con 0x7f33f8103420 2024-04-16T11:49:07.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.161+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f804c710 con 0x7f33f8103420 2024-04-16T11:49:07.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.162+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f33ec012400 con 0x7f33f8103420 2024-04-16T11:49:07.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.163+0000 7f33f5600700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f33e006ec00 0x7f33e00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:07.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.163+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f33ec08e970 con 0x7f33f8103420 2024-04-16T11:49:07.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.163+0000 7f33fd000700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f33e006ec00 0x7f33e00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:07.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.164+0000 7f33fd000700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f33e006ec00 0x7f33e00710b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f33e400b970 tx=0x7f33e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:07.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.168+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f33ec05b270 con 0x7f33f8103420 2024-04-16T11:49:07.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.425+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f33f8108a10 con 0x7f33e006ec00 2024-04-16T11:49:07.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.428+0000 7f33f5600700 1 -- 172.21.15.44:0/1446774755 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f33f8108a10 con 0x7f33e006ec00 2024-04-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f33e006ec00 msgr2=0x7f33e00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f33e006ec00 0x7f33e00710b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f33e400b970 tx=0x7f33e4009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 msgr2=0x7f33f8113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 0x7f33f8113f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c8c0 tx=0x7f33ec0044c0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 shutdown_connections 2024-04-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f33e006ec00 0x7f33e00710b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33f8103420 0x7f33f8113f50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 --2- 172.21.15.44:0/1446774755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33f81046d0 0x7f33f8114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 >> 172.21.15.44:0/1446774755 conn(0x7f33f8072640 msgr2=0x7f33f8073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:07.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 shutdown_connections 2024-04-16T11:49:07.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.432+0000 7f3400400700 1 -- 172.21.15.44:0/1446774755 wait complete. 2024-04-16T11:49:07.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.596+0000 7fa357000700 1 -- 172.21.15.44:0/1768903417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3500fffd0 msgr2=0x7fa3501023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.596+0000 7fa357000700 1 --2- 172.21.15.44:0/1768903417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3500fffd0 0x7fa3501023b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa33c004940 tx=0x7fa33c009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:07.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.597+0000 7fa357000700 1 -- 172.21.15.44:0/1768903417 shutdown_connections 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.597+0000 7fa357000700 1 --2- 172.21.15.44:0/1768903417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3500fffd0 0x7fa3501023b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.597+0000 7fa357000700 1 --2- 172.21.15.44:0/1768903417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa350066290 0x7fa3500ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.597+0000 7fa357000700 1 -- 172.21.15.44:0/1768903417 >> 172.21.15.44:0/1768903417 conn(0x7fa3500f74c0 msgr2=0x7fa3500f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.597+0000 7fa357000700 1 -- 172.21.15.44:0/1768903417 shutdown_connections 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.597+0000 7fa357000700 1 -- 172.21.15.44:0/1768903417 wait complete. 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa357000700 1 Processor -- start 2024-04-16T11:49:07.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa357000700 1 -- start start 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa357000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 0x7fa35010fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa357000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3500fffd0 0x7fa350110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa357000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa350110760 con 0x7fa350066290 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa357000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa350110fb0 con 0x7fa3500fffd0 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa34fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 0x7fa35010fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa34fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 0x7fa35010fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36096/0 (socket says 172.21.15.44:36096) 2024-04-16T11:49:07.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.598+0000 7fa34fe00700 1 -- 172.21.15.44:0/3770349302 learned_addr learned my addr 172.21.15.44:0/3770349302 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:07.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa34fe00700 1 -- 172.21.15.44:0/3770349302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3500fffd0 msgr2=0x7fa350110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa34fe00700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3500fffd0 0x7fa350110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa34fe00700 1 -- 172.21.15.44:0/3770349302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c005040 con 0x7fa350066290 2024-04-16T11:49:07.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa34fe00700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 0x7fa35010fc20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa34000c8b0 tx=0x7fa340004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:07.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa34000a760 con 0x7fa350066290 2024-04-16T11:49:07.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa34000ad90 con 0x7fa350066290 2024-04-16T11:49:07.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3400113a0 con 0x7fa350066290 2024-04-16T11:49:07.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.599+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa350111290 con 0x7fa350066290 2024-04-16T11:49:07.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.600+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3501117d0 con 0x7fa350066290 2024-04-16T11:49:07.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.601+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35005f5f0 con 0x7fa350066290 2024-04-16T11:49:07.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.602+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa340012400 con 0x7fa350066290 2024-04-16T11:49:07.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.603+0000 7fa347e00700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa33406ec10 0x7fa3340710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:07.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.603+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fa34008ea40 con 0x7fa350066290 2024-04-16T11:49:07.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.603+0000 7fa34f400700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa33406ec10 0x7fa3340710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:07.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.604+0000 7fa34f400700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa33406ec10 0x7fa3340710c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa33c004f90 tx=0x7fa33c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:07.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.609+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa34005b200 con 0x7fa350066290 2024-04-16T11:49:07.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.852+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa350111cf0 con 0x7fa33406ec10 2024-04-16T11:49:07.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.865+0000 7fa347e00700 1 -- 172.21.15.44:0/3770349302 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fa350111cf0 con 0x7fa33406ec10 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (6m) 74s ago 16m 13.2M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (3m) 74s ago 17m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (2m) 18s ago 15m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (6m) 74s ago 16m 74.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (9m) 74s ago 9m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (9m) 74s ago 9m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:49:07.863 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (9m) 18s ago 9m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (9m) 18s ago 9m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (5m) 74s ago 18m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (4m) 18s ago 15m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (4m) 74s ago 18m 46.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (4m) 18s ago 15m 34.1M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (6m) 74s ago 16m 7880k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (5m) 18s ago 15m 7931k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (2m) 74s ago 14m 50.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (112s) 74s ago 13m 42.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (82s) 74s ago 13m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (52s) 18s ago 12m 42.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (23s) 18s ago 11m 11.1M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:49:07.864 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (11m) 18s ago 11m 50.2M 2583M 17.2.7-847-g45317f95 b333581ccdf5 4db16fd1fd58 2024-04-16T11:49:07.865 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (5m) 74s ago 16m 56.6M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa33406ec10 msgr2=0x7fa3340710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa33406ec10 0x7fa3340710c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa33c004f90 tx=0x7fa33c009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 msgr2=0x7fa35010fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 0x7fa35010fc20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa34000c8b0 tx=0x7fa340004470 comp rx=0 tx=0).stop 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 shutdown_connections 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa33406ec10 0x7fa3340710c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa350066290 0x7fa35010fc20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 --2- 172.21.15.44:0/3770349302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3500fffd0 0x7fa350110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:07.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.868+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 >> 172.21.15.44:0/3770349302 conn(0x7fa3500f74c0 msgr2=0x7fa3500fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:07.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.869+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 shutdown_connections 2024-04-16T11:49:07.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:07.869+0000 7fa357000700 1 -- 172.21.15.44:0/3770349302 wait complete. 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 -- 172.21.15.44:0/1743108669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 msgr2=0x7f129c0fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 --2- 172.21.15.44:0/1743108669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c0fe460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1284004940 tx=0x7f1284009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 -- 172.21.15.44:0/1743108669 shutdown_connections 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 --2- 172.21.15.44:0/1743108669 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c0fe460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 --2- 172.21.15.44:0/1743108669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c0fce80 0x7f129c0fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 -- 172.21.15.44:0/1743108669 >> 172.21.15.44:0/1743108669 conn(0x7f129c0f8580 msgr2=0x7f129c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 -- 172.21.15.44:0/1743108669 shutdown_connections 2024-04-16T11:49:08.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.028+0000 7f12a1e00700 1 -- 172.21.15.44:0/1743108669 wait complete. 2024-04-16T11:49:08.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.029+0000 7f12a1e00700 1 Processor -- start 2024-04-16T11:49:08.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.029+0000 7f12a1e00700 1 -- start start 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f12a1e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c0fce80 0x7f129c10fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f12a1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f12a1e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c110690 con 0x7f129c0fdff0 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f12a1e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c110800 con 0x7f129c0fce80 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f1292000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f1292000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36124/0 (socket says 172.21.15.44:36124) 2024-04-16T11:49:08.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f1292000700 1 -- 172.21.15.44:0/4186497659 learned_addr learned my addr 172.21.15.44:0/4186497659 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:08.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f129aa00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c0fce80 0x7f129c10fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f1292000700 1 -- 172.21.15.44:0/4186497659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c0fce80 msgr2=0x7f129c10fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f1292000700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c0fce80 0x7f129c10fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.030+0000 7f1292000700 1 -- 172.21.15.44:0/4186497659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284005040 con 0x7f129c0fdff0 2024-04-16T11:49:08.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.031+0000 7f1292000700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c110090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f128400b3b0 tx=0x7f1284009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:08.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.031+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f128400e040 con 0x7f129c0fdff0 2024-04-16T11:49:08.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.031+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1284004290 con 0x7f129c0fdff0 2024-04-16T11:49:08.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.031+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f128401c630 con 0x7f129c0fdff0 2024-04-16T11:49:08.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.031+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c115200 con 0x7f129c0fdff0 2024-04-16T11:49:08.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.031+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c1157c0 con 0x7f129c0fdff0 2024-04-16T11:49:08.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.033+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c101980 con 0x7f129c0fdff0 2024-04-16T11:49:08.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.034+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f128400bbf0 con 0x7f129c0fdff0 2024-04-16T11:49:08.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.035+0000 7f1293400700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f128006ec00 0x7f12800710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.035+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f128408fd90 con 0x7f129c0fdff0 2024-04-16T11:49:08.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.036+0000 7f129aa00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f128006ec00 0x7f12800710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.036+0000 7f129aa00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f128006ec00 0x7f12800710b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f128c004f30 tx=0x7f128c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:08.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.040+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f128405c5d0 con 0x7f129c0fdff0 2024-04-16T11:49:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:07 smithi044 ceph-mon[93020]: pgmap v169: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:07 smithi045 ceph-mon[81423]: pgmap v169: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:08.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.360+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f129c02b3a0 con 0x7f129c0fdff0 2024-04-16T11:49:08.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.361+0000 7f1293400700 1 -- 172.21.15.44:0/4186497659 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f128405c160 con 0x7f129c0fdff0 2024-04-16T11:49:08.359 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:49:08.359 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:49:08.359 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:49:08.359 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:08.359 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 5 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 5, 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 9 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:49:08.360 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f128006ec00 msgr2=0x7f12800710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f128006ec00 0x7f12800710b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f128c004f30 tx=0x7f128c00b040 comp rx=0 tx=0).stop 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 msgr2=0x7f129c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c110090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f128400b3b0 tx=0x7f1284009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 shutdown_connections 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f128006ec00 0x7f12800710b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f129c0fce80 0x7f129c10fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 --2- 172.21.15.44:0/4186497659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f129c0fdff0 0x7f129c110090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 >> 172.21.15.44:0/4186497659 conn(0x7f129c0f8580 msgr2=0x7f129c0f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 shutdown_connections 2024-04-16T11:49:08.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.364+0000 7f12a1e00700 1 -- 172.21.15.44:0/4186497659 wait complete. 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 -- 172.21.15.44:0/1252789893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 msgr2=0x7f14b8070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 --2- 172.21.15.44:0/1252789893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8070e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14a4004940 tx=0x7f14a4009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 -- 172.21.15.44:0/1252789893 shutdown_connections 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 --2- 172.21.15.44:0/1252789893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8070e10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 --2- 172.21.15.44:0/1252789893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14b8071f50 0x7f14b8070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 -- 172.21.15.44:0/1252789893 >> 172.21.15.44:0/1252789893 conn(0x7f14b80f8530 msgr2=0x7f14b80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 -- 172.21.15.44:0/1252789893 shutdown_connections 2024-04-16T11:49:08.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.527+0000 7f14bee00700 1 -- 172.21.15.44:0/1252789893 wait complete. 2024-04-16T11:49:08.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.528+0000 7f14bee00700 1 Processor -- start 2024-04-16T11:49:08.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.528+0000 7f14bee00700 1 -- start start 2024-04-16T11:49:08.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.528+0000 7f14bee00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.528+0000 7f14bee00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14b8071f50 0x7f14b8114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14bee00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b8114b80 con 0x7f14b80709a0 2024-04-16T11:49:08.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14bee00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b81153d0 con 0x7f14b8071f50 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36152/0 (socket says 172.21.15.44:36152) 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7e00700 1 -- 172.21.15.44:0/926183201 learned_addr learned my addr 172.21.15.44:0/926183201 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7e00700 1 -- 172.21.15.44:0/926183201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14b8071f50 msgr2=0x7f14b8114580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7400700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14b8071f50 0x7f14b8114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7e00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14b8071f50 0x7f14b8114580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.529+0000 7f14b7e00700 1 -- 172.21.15.44:0/926183201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4005040 con 0x7f14b80709a0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.530+0000 7f14b7e00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8114040 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14a8002a80 tx=0x7f14a8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.530+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a8007cc0 con 0x7f14b80709a0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.530+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14a800f040 con 0x7f14b80709a0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.530+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a800dad0 con 0x7f14b80709a0 2024-04-16T11:49:08.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.530+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b81156b0 con 0x7f14b80709a0 2024-04-16T11:49:08.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.530+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b8115bf0 con 0x7f14b80709a0 2024-04-16T11:49:08.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.531+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b805f5f0 con 0x7f14b80709a0 2024-04-16T11:49:08.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.532+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f14a800dc30 con 0x7f14b80709a0 2024-04-16T11:49:08.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.533+0000 7f14afe00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f149c06ec10 0x7f149c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.533+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f14a808e720 con 0x7f14b80709a0 2024-04-16T11:49:08.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.534+0000 7f14b7400700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f149c06ec10 0x7f149c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.534+0000 7f14b7400700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f149c06ec10 0x7f149c0710c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14a400b970 tx=0x7f14a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:08.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.538+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f14a805aee0 con 0x7f14b80709a0 2024-04-16T11:49:08.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.822+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f14b804bed0 con 0x7f14b80709a0 2024-04-16T11:49:08.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.823+0000 7f14afe00700 1 -- 172.21.15.44:0/926183201 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f14a805aa70 con 0x7f14b80709a0 2024-04-16T11:49:08.821 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:49:08.821 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:49:08.821 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:49:08.822 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:49:08.823 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:08.824 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.825+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f149c06ec10 msgr2=0x7f149c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.825+0000 7f14bee00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f149c06ec10 0x7f149c0710c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14a400b970 tx=0x7f14a4009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 msgr2=0x7f14b8114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8114040 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14a8002a80 tx=0x7f14a8004600 comp rx=0 tx=0).stop 2024-04-16T11:49:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 shutdown_connections 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f149c06ec10 0x7f149c0710c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f14b80709a0 0x7f14b8114040 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 --2- 172.21.15.44:0/926183201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14b8071f50 0x7f14b8114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 >> 172.21.15.44:0/926183201 conn(0x7f14b80f8530 msgr2=0x7f14b80f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 shutdown_connections 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.826+0000 7f14bee00700 1 -- 172.21.15.44:0/926183201 wait complete. 2024-04-16T11:49:08.826 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:49:08.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.979+0000 7f78e6e00700 1 -- 172.21.15.44:0/376062476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 msgr2=0x7f78e0100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.979+0000 7f78e6e00700 1 --2- 172.21.15.44:0/376062476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e0100740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f78c8004940 tx=0x7f78c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:08.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 -- 172.21.15.44:0/376062476 shutdown_connections 2024-04-16T11:49:08.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 --2- 172.21.15.44:0/376062476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e0100740 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 --2- 172.21.15.44:0/376062476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78e00ff020 0x7f78e00ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 -- 172.21.15.44:0/376062476 >> 172.21.15.44:0/376062476 conn(0x7f78e00fa950 msgr2=0x7f78e00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:08.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 -- 172.21.15.44:0/376062476 shutdown_connections 2024-04-16T11:49:08.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 -- 172.21.15.44:0/376062476 wait complete. 2024-04-16T11:49:08.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.980+0000 7f78e6e00700 1 Processor -- start 2024-04-16T11:49:08.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78e6e00700 1 -- start start 2024-04-16T11:49:08.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78e6e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78e00ff020 0x7f78e010fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78e6e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e01100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78e6e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e01106c0 con 0x7f78e01002d0 2024-04-16T11:49:08.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78e6e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e0110830 con 0x7f78e00ff020 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e01100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e01100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36176/0 (socket says 172.21.15.44:36176) 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.981+0000 7f78d7e00700 1 -- 172.21.15.44:0/3790238725 learned_addr learned my addr 172.21.15.44:0/3790238725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78dfe00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78e00ff020 0x7f78e010fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78d7e00700 1 -- 172.21.15.44:0/3790238725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78e00ff020 msgr2=0x7f78e010fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78d7e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78e00ff020 0x7f78e010fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:08.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78d7e00700 1 -- 172.21.15.44:0/3790238725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8005040 con 0x7f78e01002d0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78d7e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e01100c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f78c800b3b0 tx=0x7f78c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78c800e040 con 0x7f78e01002d0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e0115230 con 0x7f78e01002d0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78c8004290 con 0x7f78e01002d0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.982+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78c801c630 con 0x7f78e01002d0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.983+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e01157f0 con 0x7f78e01002d0 2024-04-16T11:49:08.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.983+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e004c710 con 0x7f78e01002d0 2024-04-16T11:49:08.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.984+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f78c800bbf0 con 0x7f78e01002d0 2024-04-16T11:49:08.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.985+0000 7f78dcc00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f78c406ec00 0x7f78c40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:08.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.986+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f78c808fd90 con 0x7f78e01002d0 2024-04-16T11:49:08.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.986+0000 7f78dfe00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f78c406ec00 0x7f78c40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:08.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.986+0000 7f78dfe00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f78c406ec00 0x7f78c40710b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f78d0007fd0 tx=0x7f78d0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:08.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:08.991+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f78c805c5d0 con 0x7f78e01002d0 2024-04-16T11:49:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:08 smithi044 ceph-mon[93020]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:08 smithi044 ceph-mon[93020]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:08 smithi044 ceph-mon[93020]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:08 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/4186497659' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:08 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/926183201' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:49:09.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.233+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78e01045f0 con 0x7f78c406ec00 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.236+0000 7f78dcc00700 1 -- 172.21.15.44:0/3790238725 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f78e01045f0 con 0x7f78c406ec00 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:49:09.234 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "11/21 daemons upgraded", 2024-04-16T11:49:09.235 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T11:49:09.235 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:49:09.235 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f78c406ec00 msgr2=0x7f78c40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f78c406ec00 0x7f78c40710b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f78d0007fd0 tx=0x7f78d0009040 comp rx=0 tx=0).stop 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 msgr2=0x7f78e01100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e01100c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f78c800b3b0 tx=0x7f78c8009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 shutdown_connections 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f78c406ec00 0x7f78c40710b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78e00ff020 0x7f78e010fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 --2- 172.21.15.44:0/3790238725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78e01002d0 0x7f78e01100c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.238+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 >> 172.21.15.44:0/3790238725 conn(0x7f78e00fa950 msgr2=0x7f78e00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:09.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.239+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 shutdown_connections 2024-04-16T11:49:09.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.239+0000 7f78e6e00700 1 -- 172.21.15.44:0/3790238725 wait complete. 2024-04-16T11:49:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:08 smithi045 ceph-mon[81423]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:08 smithi045 ceph-mon[81423]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:08 smithi045 ceph-mon[81423]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:08 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/4186497659' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:09.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:08 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/926183201' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:49:09.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.396+0000 7f4c0ac00700 1 -- 172.21.15.44:0/3340127872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c040709a0 msgr2=0x7f4c04070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:09.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.396+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/3340127872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c040709a0 0x7f4c04070e10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4bec004940 tx=0x7f4bec009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.396+0000 7f4c0ac00700 1 -- 172.21.15.44:0/3340127872 shutdown_connections 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.396+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/3340127872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c040709a0 0x7f4c04070e10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.396+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/3340127872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c04071f50 0x7f4c04070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.396+0000 7f4c0ac00700 1 -- 172.21.15.44:0/3340127872 >> 172.21.15.44:0/3340127872 conn(0x7f4c040f8530 msgr2=0x7f4c040fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.397+0000 7f4c0ac00700 1 -- 172.21.15.44:0/3340127872 shutdown_connections 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.397+0000 7f4c0ac00700 1 -- 172.21.15.44:0/3340127872 wait complete. 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.397+0000 7f4c0ac00700 1 Processor -- start 2024-04-16T11:49:09.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4c0ac00700 1 -- start start 2024-04-16T11:49:09.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4c0ac00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c040709a0 0x7f4c04114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:09.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4c0ac00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 0x7f4c04114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:09.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4c0ac00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c04114b80 con 0x7f4c04071f50 2024-04-16T11:49:09.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4c0ac00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c041153d0 con 0x7f4c040709a0 2024-04-16T11:49:09.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4bfbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 0x7f4c04114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:09.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4bfbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 0x7f4c04114580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36194/0 (socket says 172.21.15.44:36194) 2024-04-16T11:49:09.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.398+0000 7f4bfbe00700 1 -- 172.21.15.44:0/232761365 learned_addr learned my addr 172.21.15.44:0/232761365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:09.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.399+0000 7f4c03e00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c040709a0 0x7f4c04114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:09.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.399+0000 7f4bfbe00700 1 -- 172.21.15.44:0/232761365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c040709a0 msgr2=0x7f4c04114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.399+0000 7f4bfbe00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c040709a0 0x7f4c04114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.399+0000 7f4bfbe00700 1 -- 172.21.15.44:0/232761365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec005040 con 0x7f4c04071f50 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.399+0000 7f4bfbe00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 0x7f4c04114580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00b3b0 tx=0x7f4bec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.400+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bec00e040 con 0x7f4c04071f50 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.400+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bec004290 con 0x7f4c04071f50 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.400+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c04115650 con 0x7f4c04071f50 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.400+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bec01c630 con 0x7f4c04071f50 2024-04-16T11:49:09.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.400+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c04115b90 con 0x7f4c04071f50 2024-04-16T11:49:09.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.401+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0405f5f0 con 0x7f4c04071f50 2024-04-16T11:49:09.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.402+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4bec00bbf0 con 0x7f4c04071f50 2024-04-16T11:49:09.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.404+0000 7f4c00c00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4be806ec10 0x7f4be80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:09.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.404+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f4bec08fdc0 con 0x7f4c04071f50 2024-04-16T11:49:09.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.404+0000 7f4c03e00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4be806ec10 0x7f4be80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:09.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.405+0000 7f4c03e00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4be806ec10 0x7f4be80710c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4007fd0 tx=0x7f4bf4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:09.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.409+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4bec05c620 con 0x7f4c04071f50 2024-04-16T11:49:09.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.736+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4c0410bf90 con 0x7f4c04071f50 2024-04-16T11:49:09.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.737+0000 7f4c00c00700 1 -- 172.21.15.44:0/232761365 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f4bec05c1b0 con 0x7f4c04071f50 2024-04-16T11:49:09.735 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:49:09.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4be806ec10 msgr2=0x7f4be80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:09.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4be806ec10 0x7f4be80710c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4007fd0 tx=0x7f4bf4009040 comp rx=0 tx=0).stop 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 msgr2=0x7f4c04114580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 0x7f4c04114580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00b3b0 tx=0x7f4bec009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 shutdown_connections 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4be806ec10 0x7f4be80710c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c040709a0 0x7f4c04114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 --2- 172.21.15.44:0/232761365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c04071f50 0x7f4c04114580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.740+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 >> 172.21.15.44:0/232761365 conn(0x7f4c040f8530 msgr2=0x7f4c040f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.741+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 shutdown_connections 2024-04-16T11:49:09.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:09.741+0000 7f4c0ac00700 1 -- 172.21.15.44:0/232761365 wait complete. 2024-04-16T11:49:10.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:09 smithi044 ceph-mon[93020]: pgmap v170: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:10.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:09 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/232761365' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:49:10.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:09 smithi045 ceph-mon[81423]: pgmap v170: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:10.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:09 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/232761365' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:49:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:10 smithi044 ceph-mon[93020]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:10 smithi045 ceph-mon[81423]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:12.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:11 smithi044 ceph-mon[93020]: pgmap v171: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:12.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:11 smithi045 ceph-mon[81423]: pgmap v171: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:13 smithi044 ceph-mon[93020]: pgmap v172: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:13 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:14.247 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:13 smithi045 ceph-mon[81423]: pgmap v172: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:14.247 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:13 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:15.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:14 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:14 smithi045 ceph-mon[81423]: Upgrade: osd.5 is safe to restart 2024-04-16T11:49:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:14 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T11:49:15.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:14 smithi044 ceph-mon[93020]: Upgrade: osd.5 is safe to restart 2024-04-16T11:49:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:15 smithi044 ceph-mon[93020]: pgmap v173: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T11:49:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:49:16.288 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:15 smithi045 ceph-mon[81423]: pgmap v173: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:16.288 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:16.288 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T11:49:16.288 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:49:17.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:16 smithi044 ceph-mon[93020]: Upgrade: Updating osd.5 2024-04-16T11:49:17.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:16 smithi044 ceph-mon[93020]: Deploying daemon osd.5 on smithi045 2024-04-16T11:49:17.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:16 smithi044 ceph-mon[93020]: pgmap v174: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:16 smithi045 ceph-mon[81423]: Upgrade: Updating osd.5 2024-04-16T11:49:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:16 smithi045 ceph-mon[81423]: Deploying daemon osd.5 on smithi045 2024-04-16T11:49:17.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:16 smithi045 ceph-mon[81423]: pgmap v174: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:19.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:19 smithi045 ceph-mon[81423]: pgmap v175: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:19.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:19 smithi044 ceph-mon[93020]: pgmap v175: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:20.561 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:20 smithi045 systemd[1]: Stopping Ceph osd.5 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:49:20.829 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:20 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[55147]: 2024-04-16T11:49:20.563+0000 7fec6e600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T11:49:20.830 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:20 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[55147]: 2024-04-16T11:49:20.563+0000 7fec6e600700 -1 osd.5 81 *** Got signal Terminated *** 2024-04-16T11:49:20.830 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:20 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[55147]: 2024-04-16T11:49:20.563+0000 7fec6e600700 -1 osd.5 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T11:49:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:21 smithi045 ceph-mon[81423]: pgmap v176: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:21.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:21 smithi045 ceph-mon[81423]: osd.5 marked itself down and dead 2024-04-16T11:49:21.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:21 smithi044 ceph-mon[93020]: pgmap v176: 65 pgs: 65 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:21.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:21 smithi044 ceph-mon[93020]: osd.5 marked itself down and dead 2024-04-16T11:49:22.060 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:21 smithi045 podman[100882]: 2024-04-16 11:49:21.788278952 +0000 UTC m=+1.346208449 container died 4db16fd1fd582588fd60aed1b270bb5e1fe90e3f473711f2c23fd6edc35a1577 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5) 2024-04-16T11:49:22.315 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:22 smithi045 podman[100882]: 2024-04-16 11:49:22.239549343 +0000 UTC m=+1.797478897 container cleanup 4db16fd1fd582588fd60aed1b270bb5e1fe90e3f473711f2c23fd6edc35a1577 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T11:49:22.315 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:22 smithi045 bash[100882]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5 2024-04-16T11:49:22.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:22 smithi045 ceph-mon[81423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:49:22.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:22 smithi045 ceph-mon[81423]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T11:49:22.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:22 smithi045 ceph-mon[81423]: osdmap e82: 6 total, 5 up, 6 in 2024-04-16T11:49:22.580 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:22 smithi045 podman[100901]: 2024-04-16 11:49:22.441040451 +0000 UTC m=+0.635752396 container remove 4db16fd1fd582588fd60aed1b270bb5e1fe90e3f473711f2c23fd6edc35a1577 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-16T11:49:22.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:22 smithi044 ceph-mon[93020]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T11:49:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:22 smithi044 ceph-mon[93020]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T11:49:22.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:22 smithi044 ceph-mon[93020]: osdmap e82: 6 total, 5 up, 6 in 2024-04-16T11:49:23.551 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:23 smithi045 ceph-mon[81423]: pgmap v178: 65 pgs: 14 stale+active+clean, 51 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:23.551 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:23 smithi045 ceph-mon[81423]: osdmap e83: 6 total, 5 up, 6 in 2024-04-16T11:49:23.552 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:23 smithi045 podman[100986]: 2024-04-16 11:49:23.292073766 +0000 UTC m=+0.131051826 container create 3aefe4098604c1db20a61eb3cfecdf4852a3d2ad2b9f772370550119b82ad5f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-16T11:49:23.552 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:23 smithi045 podman[100986]: 2024-04-16 11:49:23.198994246 +0000 UTC m=+0.037972282 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:49:23.552 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:23 smithi045 podman[100986]: 2024-04-16 11:49:23.553968427 +0000 UTC m=+0.392946494 container init 3aefe4098604c1db20a61eb3cfecdf4852a3d2ad2b9f772370550119b82ad5f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T11:49:23.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:23 smithi044 ceph-mon[93020]: pgmap v178: 65 pgs: 14 stale+active+clean, 51 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:23.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:23 smithi044 ceph-mon[93020]: osdmap e83: 6 total, 5 up, 6 in 2024-04-16T11:49:23.829 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:23 smithi045 podman[100986]: 2024-04-16 11:49:23.56760414 +0000 UTC m=+0.406582230 container start 3aefe4098604c1db20a61eb3cfecdf4852a3d2ad2b9f772370550119b82ad5f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-16T11:49:23.829 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:23 smithi045 podman[100986]: 2024-04-16 11:49:23.670661974 +0000 UTC m=+0.509640041 container attach 3aefe4098604c1db20a61eb3cfecdf4852a3d2ad2b9f772370550119b82ad5f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T11:49:24.330 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:24 smithi045 podman[100986]: 2024-04-16 11:49:24.034404854 +0000 UTC m=+0.873382927 container died 3aefe4098604c1db20a61eb3cfecdf4852a3d2ad2b9f772370550119b82ad5f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-16T11:49:25.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:24 smithi045 podman[101012]: 2024-04-16 11:49:24.726842055 +0000 UTC m=+0.671621694 container remove 3aefe4098604c1db20a61eb3cfecdf4852a3d2ad2b9f772370550119b82ad5f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-16T11:49:25.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:24 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-16T11:49:25.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:24 smithi045 systemd[1]: Stopped Ceph osd.5 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:49:25.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:24 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.5.service: Consumed 14.042s CPU time. 2024-04-16T11:49:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:25 smithi044 ceph-mon[93020]: pgmap v180: 65 pgs: 11 stale+active+clean, 5 active+undersized, 49 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:25.559 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:25 smithi045 systemd[1]: Starting Ceph osd.5 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T11:49:25.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:25 smithi045 ceph-mon[81423]: pgmap v180: 65 pgs: 11 stale+active+clean, 5 active+undersized, 49 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:25.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:25.560 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:25.826 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:25 smithi045 podman[101118]: 2024-04-16 11:49:25.467883109 +0000 UTC m=+0.036016025 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:49:25.826 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:25 smithi045 podman[101118]: 2024-04-16 11:49:25.582284994 +0000 UTC m=+0.150417947 container create 33a80fb40b4b0951f05c9c6ab20e3897b41592d1dce1b197dbf2f3c1d193f028 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-16T11:49:25.826 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:25 smithi045 podman[101118]: 2024-04-16 11:49:25.828038365 +0000 UTC m=+0.396171341 container init 33a80fb40b4b0951f05c9c6ab20e3897b41592d1dce1b197dbf2f3c1d193f028 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T11:49:26.079 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:25 smithi045 podman[101118]: 2024-04-16 11:49:25.841050001 +0000 UTC m=+0.409182926 container start 33a80fb40b4b0951f05c9c6ab20e3897b41592d1dce1b197dbf2f3c1d193f028 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-16T11:49:26.079 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:25 smithi045 podman[101118]: 2024-04-16 11:49:25.918168414 +0000 UTC m=+0.486301374 container attach 33a80fb40b4b0951f05c9c6ab20e3897b41592d1dce1b197dbf2f3c1d193f028 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-16T11:49:26.628 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 464, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101219 2024-04-16T11:49:26.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 464, success, outsize: 120 2024-04-16T11:49:26.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:26 smithi044 ceph-mon[93020]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:49:26.758 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-mon[81423]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T11:49:27.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T11:49:27.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T11:49:27.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T11:49:27.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T11:49:27.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T11:49:27.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 bash[101118]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate[101132]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T11:49:27.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:26 smithi045 podman[101118]: 2024-04-16 11:49:26.825929531 +0000 UTC m=+1.394062527 container died 33a80fb40b4b0951f05c9c6ab20e3897b41592d1dce1b197dbf2f3c1d193f028 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-16T11:49:27.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:27 smithi044 ceph-mon[93020]: pgmap v181: 65 pgs: 5 active+undersized+degraded, 3 stale+active+clean, 19 active+undersized, 38 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:27.792 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:27 smithi045 ceph-mon[81423]: pgmap v181: 65 pgs: 5 active+undersized+degraded, 3 stale+active+clean, 19 active+undersized, 38 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:27.792 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:27 smithi045 podman[101257]: 2024-04-16 11:49:27.429609746 +0000 UTC m=+0.583390688 container remove 33a80fb40b4b0951f05c9c6ab20e3897b41592d1dce1b197dbf2f3c1d193f028 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T11:49:28.079 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:27 smithi045 podman[101319]: 2024-04-16 11:49:27.702597215 +0000 UTC m=+0.037781315 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T11:49:28.080 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:27 smithi045 podman[101319]: 2024-04-16 11:49:27.822011346 +0000 UTC m=+0.157195447 container create 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T11:49:28.580 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:28 smithi045 podman[101319]: 2024-04-16 11:49:28.085290876 +0000 UTC m=+0.420475001 container init 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-16T11:49:28.580 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:28 smithi045 podman[101319]: 2024-04-16 11:49:28.097291221 +0000 UTC m=+0.432475323 container start 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-16T11:49:28.580 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:28 smithi045 ceph-osd[101338]: -- 172.21.15.45:0/3606815728 <== mon.1 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a3a3b57680 con 0x55a3a3b21000 2024-04-16T11:49:28.580 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:28 smithi045 bash[101319]: 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a 2024-04-16T11:49:28.580 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:28 smithi045 systemd[1]: Started Ceph osd.5 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T11:49:29.079 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:28 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T11:49:28.690+0000 7f2f358fa7c0 -1 Falling back to public interface 2024-04-16T11:49:29.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:29 smithi045 ceph-mon[81423]: pgmap v182: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:29.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:29.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:49:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:29 smithi044 ceph-mon[93020]: pgmap v182: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:49:31.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:31 smithi045 ceph-mon[81423]: pgmap v183: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:31.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:31 smithi044 ceph-mon[93020]: pgmap v183: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:32.829 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:32 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T11:49:32.350+0000 7f2f358fa7c0 -1 osd.5 81 log_to_monitors true 2024-04-16T11:49:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:33 smithi044 ceph-mon[93020]: pgmap v184: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:33 smithi044 ceph-mon[93020]: from='osd.5 [v2:172.21.15.45:6816/579419871,v1:172.21.15.45:6817/579419871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:49:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:33 smithi044 ceph-mon[93020]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:49:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:33 smithi045 ceph-mon[81423]: pgmap v184: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 440 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:33 smithi045 ceph-mon[81423]: from='osd.5 [v2:172.21.15.45:6816/579419871,v1:172.21.15.45:6817/579419871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:49:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:33 smithi045 ceph-mon[81423]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T11:49:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:34 smithi044 ceph-mon[93020]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T11:49:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:34 smithi044 ceph-mon[93020]: from='osd.5 [v2:172.21.15.45:6816/579419871,v1:172.21.15.45:6817/579419871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:49:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:34 smithi044 ceph-mon[93020]: osdmap e84: 6 total, 5 up, 6 in 2024-04-16T11:49:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:34 smithi044 ceph-mon[93020]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:49:34.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:34 smithi045 ceph-mon[81423]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T11:49:34.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:34 smithi045 ceph-mon[81423]: from='osd.5 [v2:172.21.15.45:6816/579419871,v1:172.21.15.45:6817/579419871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:49:34.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:34 smithi045 ceph-mon[81423]: osdmap e84: 6 total, 5 up, 6 in 2024-04-16T11:49:34.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:34 smithi045 ceph-mon[81423]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-16T11:49:34.731 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 11:49:34 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T11:49:34.377+0000 7f2f1fc00700 -1 osd.5 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T11:49:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:35 smithi044 ceph-mon[93020]: pgmap v186: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:35 smithi044 ceph-mon[93020]: from='osd.5 ' entity='osd.5' 2024-04-16T11:49:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:35.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:35.690 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:35 smithi045 ceph-mon[81423]: pgmap v186: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T11:49:35.690 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:35 smithi045 ceph-mon[81423]: from='osd.5 ' entity='osd.5' 2024-04-16T11:49:35.690 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:35.690 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:36.702 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:36 smithi045 ceph-mon[81423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:49:36.702 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:36 smithi045 ceph-mon[81423]: osd.5 [v2:172.21.15.45:6816/579419871,v1:172.21.15.45:6817/579419871] boot 2024-04-16T11:49:36.702 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:36 smithi045 ceph-mon[81423]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T11:49:36.702 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:49:36.702 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:36 smithi045 ceph-mon[81423]: osdmap e86: 6 total, 6 up, 6 in 2024-04-16T11:49:36.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:36 smithi044 ceph-mon[93020]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T11:49:36.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:36 smithi044 ceph-mon[93020]: osd.5 [v2:172.21.15.45:6816/579419871,v1:172.21.15.45:6817/579419871] boot 2024-04-16T11:49:36.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:36 smithi044 ceph-mon[93020]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T11:49:36.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T11:49:36.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:36 smithi044 ceph-mon[93020]: osdmap e86: 6 total, 6 up, 6 in 2024-04-16T11:49:37.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:37 smithi044 ceph-mon[93020]: pgmap v188: 65 pgs: 9 peering, 23 active+undersized, 33 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:37.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:37 smithi044 ceph-mon[93020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded) 2024-04-16T11:49:37.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:37 smithi045 ceph-mon[81423]: pgmap v188: 65 pgs: 9 peering, 23 active+undersized, 33 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:37.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:37 smithi045 ceph-mon[81423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded) 2024-04-16T11:49:38.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 466, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103053 2024-04-16T11:49:38.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 466, success, outsize: 120 2024-04-16T11:49:38.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 468, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103136 2024-04-16T11:49:38.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 468, success, outsize: 120 2024-04-16T11:49:38.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 470, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103222 2024-04-16T11:49:38.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 470, success, outsize: 120 2024-04-16T11:49:38.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 472, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103314 2024-04-16T11:49:38.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 472, success, outsize: 120 2024-04-16T11:49:38.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 474, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103403 2024-04-16T11:49:38.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 474, success, outsize: 120 2024-04-16T11:49:38.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 476, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103486 2024-04-16T11:49:38.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 476, success, outsize: 120 2024-04-16T11:49:38.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 478, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103572 2024-04-16T11:49:38.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 478, success, outsize: 120 2024-04-16T11:49:38.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 480, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103664 2024-04-16T11:49:38.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 480, success, outsize: 120 2024-04-16T11:49:38.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 482, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103753 2024-04-16T11:49:38.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 482, success, outsize: 120 2024-04-16T11:49:38.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 484, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103836 2024-04-16T11:49:38.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 484, success, outsize: 120 2024-04-16T11:49:38.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 486, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103922 2024-04-16T11:49:38.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 486, success, outsize: 120 2024-04-16T11:49:38.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 488, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104014 2024-04-16T11:49:38.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 488, success, outsize: 120 2024-04-16T11:49:38.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 490, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104103 2024-04-16T11:49:38.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 490, success, outsize: 120 2024-04-16T11:49:38.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 492, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104186 2024-04-16T11:49:38.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 492, success, outsize: 120 2024-04-16T11:49:38.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 494, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104272 2024-04-16T11:49:38.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 494, success, outsize: 120 2024-04-16T11:49:38.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 496, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104364 2024-04-16T11:49:38.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 496, success, outsize: 120 2024-04-16T11:49:38.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 498, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104453 2024-04-16T11:49:38.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 498, success, outsize: 120 2024-04-16T11:49:38.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 500, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104536 2024-04-16T11:49:38.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 500, success, outsize: 120 2024-04-16T11:49:38.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 502, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104622 2024-04-16T11:49:38.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 502, success, outsize: 120 2024-04-16T11:49:38.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 504, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104714 2024-04-16T11:49:38.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 504, success, outsize: 120 2024-04-16T11:49:39.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 506, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104803 2024-04-16T11:49:39.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 506, success, outsize: 120 2024-04-16T11:49:39.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 508, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104886 2024-04-16T11:49:39.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 508, success, outsize: 120 2024-04-16T11:49:39.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 510, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104972 2024-04-16T11:49:39.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 510, success, outsize: 120 2024-04-16T11:49:39.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 512, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105064 2024-04-16T11:49:39.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 512, success, outsize: 120 2024-04-16T11:49:39.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 514, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105153 2024-04-16T11:49:39.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 514, success, outsize: 120 2024-04-16T11:49:39.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 516, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105236 2024-04-16T11:49:39.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 516, success, outsize: 120 2024-04-16T11:49:39.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 518, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105322 2024-04-16T11:49:39.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 518, success, outsize: 120 2024-04-16T11:49:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 520, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105414 2024-04-16T11:49:39.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 520, success, outsize: 120 2024-04-16T11:49:39.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:39 smithi045 ceph-mon[81423]: pgmap v190: 65 pgs: 14 peering, 18 active+undersized, 33 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:39.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:39 smithi044 ceph-mon[93020]: pgmap v190: 65 pgs: 14 peering, 18 active+undersized, 33 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.917+0000 7f03cfe00700 1 -- 172.21.15.44:0/1506629527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 msgr2=0x7f03d0070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.917+0000 7f03cfe00700 1 --2- 172.21.15.44:0/1506629527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d0070350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f03c0004940 tx=0x7f03c0009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.918+0000 7f03cfe00700 1 -- 172.21.15.44:0/1506629527 shutdown_connections 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.918+0000 7f03cfe00700 1 --2- 172.21.15.44:0/1506629527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d0070890 0x7f03d0070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.918+0000 7f03cfe00700 1 --2- 172.21.15.44:0/1506629527 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d0070350 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.918+0000 7f03cfe00700 1 -- 172.21.15.44:0/1506629527 >> 172.21.15.44:0/1506629527 conn(0x7f03d00f83a0 msgr2=0x7f03d00fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.918+0000 7f03cfe00700 1 -- 172.21.15.44:0/1506629527 shutdown_connections 2024-04-16T11:49:39.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.918+0000 7f03cfe00700 1 -- 172.21.15.44:0/1506629527 wait complete. 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03cfe00700 1 Processor -- start 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03cfe00700 1 -- start start 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03cfe00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d0070890 0x7f03d010fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03cfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d010ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03cfe00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0110590 con 0x7f03d0071e40 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03cfe00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0110de0 con 0x7f03d0070890 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.919+0000 7f03ce000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d010ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03ce000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d010ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53084/0 (socket says 172.21.15.44:53084) 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03ce000700 1 -- 172.21.15.44:0/809467369 learned_addr learned my addr 172.21.15.44:0/809467369 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:39.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03cea00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d0070890 0x7f03d010fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03ce000700 1 -- 172.21.15.44:0/809467369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d0070890 msgr2=0x7f03d010fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03ce000700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d0070890 0x7f03d010fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03ce000700 1 -- 172.21.15.44:0/809467369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0005040 con 0x7f03d0071e40 2024-04-16T11:49:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.920+0000 7f03ce000700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d010ff90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f03bc002a50 tx=0x7f03bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.921+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc007cc0 con 0x7f03d0071e40 2024-04-16T11:49:39.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.921+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03bc00f040 con 0x7f03d0071e40 2024-04-16T11:49:39.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.921+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d01110c0 con 0x7f03d0071e40 2024-04-16T11:49:39.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.921+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc009640 con 0x7f03d0071e40 2024-04-16T11:49:39.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.921+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d0111600 con 0x7f03d0071e40 2024-04-16T11:49:39.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.922+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d00fc020 con 0x7f03d0071e40 2024-04-16T11:49:39.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.923+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f03bc0097a0 con 0x7f03d0071e40 2024-04-16T11:49:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.924+0000 7f03c6a00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03b406ec00 0x7f03b40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.924+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f03bc08e750 con 0x7f03d0071e40 2024-04-16T11:49:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.924+0000 7f03cea00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03b406ec00 0x7f03b40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.925+0000 7f03cea00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03b406ec00 0x7f03b40710b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b3b0 tx=0x7f03c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:39.930+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f03bc05af10 con 0x7f03d0071e40 2024-04-16T11:49:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.188+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03d0111a30 con 0x7f03b406ec00 2024-04-16T11:49:40.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.191+0000 7f03c6a00700 1 -- 172.21.15.44:0/809467369 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f03d0111a30 con 0x7f03b406ec00 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03b406ec00 msgr2=0x7f03b40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03b406ec00 0x7f03b40710b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b3b0 tx=0x7f03c0009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 msgr2=0x7f03d010ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d010ff90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f03bc002a50 tx=0x7f03bc0046b0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 shutdown_connections 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03b406ec00 0x7f03b40710b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d0070890 0x7f03d010fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 --2- 172.21.15.44:0/809467369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d0071e40 0x7f03d010ff90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 >> 172.21.15.44:0/809467369 conn(0x7f03d00f83a0 msgr2=0x7f03d00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 shutdown_connections 2024-04-16T11:49:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.194+0000 7f03cfe00700 1 -- 172.21.15.44:0/809467369 wait complete. 2024-04-16T11:49:40.206 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.340+0000 7f9879400700 1 -- 172.21.15.44:0/2447180023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98741046d0 msgr2=0x7f9874104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.340+0000 7f9879400700 1 --2- 172.21.15.44:0/2447180023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98741046d0 0x7f9874104b40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9860004940 tx=0x7f9860009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 -- 172.21.15.44:0/2447180023 shutdown_connections 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 --2- 172.21.15.44:0/2447180023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98741046d0 0x7f9874104b40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 --2- 172.21.15.44:0/2447180023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9874103420 0x7f9874103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 -- 172.21.15.44:0/2447180023 >> 172.21.15.44:0/2447180023 conn(0x7f9874072640 msgr2=0x7f9874074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 -- 172.21.15.44:0/2447180023 shutdown_connections 2024-04-16T11:49:40.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 -- 172.21.15.44:0/2447180023 wait complete. 2024-04-16T11:49:40.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.341+0000 7f9879400700 1 Processor -- start 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9879400700 1 -- start start 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9879400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 0x7f9874113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9879400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98741046d0 0x7f9874114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9879400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874114a90 con 0x7f9874103420 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9879400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874114c00 con 0x7f98741046d0 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9872800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 0x7f9874113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9872800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 0x7f9874113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53110/0 (socket says 172.21.15.44:53110) 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9872800700 1 -- 172.21.15.44:0/508853083 learned_addr learned my addr 172.21.15.44:0/508853083 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.342+0000 7f9871e00700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98741046d0 0x7f9874114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.343+0000 7f9872800700 1 -- 172.21.15.44:0/508853083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98741046d0 msgr2=0x7f9874114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.343+0000 7f9872800700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98741046d0 0x7f9874114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.343+0000 7f9872800700 1 -- 172.21.15.44:0/508853083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9860005040 con 0x7f9874103420 2024-04-16T11:49:40.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.343+0000 7f9872800700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 0x7f9874113f50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9864002a50 tx=0x7f98640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:40.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.343+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9864007cc0 con 0x7f9874103420 2024-04-16T11:49:40.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.344+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f986400f040 con 0x7f9874103420 2024-04-16T11:49:40.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.344+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f986400da30 con 0x7f9874103420 2024-04-16T11:49:40.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.344+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9874119660 con 0x7f9874103420 2024-04-16T11:49:40.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.344+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9874119c20 con 0x7f9874103420 2024-04-16T11:49:40.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.345+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987404c710 con 0x7f9874103420 2024-04-16T11:49:40.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.346+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f986400db90 con 0x7f9874103420 2024-04-16T11:49:40.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.347+0000 7f986aa00700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f985806ec00 0x7f98580710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:40.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.347+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f986408e650 con 0x7f9874103420 2024-04-16T11:49:40.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.347+0000 7f9871e00700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f985806ec00 0x7f98580710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:40.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.348+0000 7f9871e00700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f985806ec00 0x7f98580710b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9860009fd0 tx=0x7f9860009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:40.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.353+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f986405aeb0 con 0x7f9874103420 2024-04-16T11:49:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.622+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9874108a10 con 0x7f985806ec00 2024-04-16T11:49:40.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.625+0000 7f986aa00700 1 -- 172.21.15.44:0/508853083 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9874108a10 con 0x7f985806ec00 2024-04-16T11:49:40.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f985806ec00 msgr2=0x7f98580710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f985806ec00 0x7f98580710b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9860009fd0 tx=0x7f9860009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:40.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 msgr2=0x7f9874113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 0x7f9874113f50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9864002a50 tx=0x7f98640046b0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 shutdown_connections 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f985806ec00 0x7f98580710b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9874103420 0x7f9874113f50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 --2- 172.21.15.44:0/508853083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98741046d0 0x7f9874114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.627+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 >> 172.21.15.44:0/508853083 conn(0x7f9874072640 msgr2=0x7f9874073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.628+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 shutdown_connections 2024-04-16T11:49:40.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.628+0000 7f9879400700 1 -- 172.21.15.44:0/508853083 wait complete. 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.785+0000 7f894c200700 1 -- 172.21.15.44:0/20065690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 msgr2=0x7f89440fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.785+0000 7f894c200700 1 --2- 172.21.15.44:0/20065690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f89440fd1d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8938004940 tx=0x7f8938009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.786+0000 7f894c200700 1 -- 172.21.15.44:0/20065690 shutdown_connections 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.786+0000 7f894c200700 1 --2- 172.21.15.44:0/20065690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 0x7f89440fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.786+0000 7f894c200700 1 --2- 172.21.15.44:0/20065690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f89440fd1d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.786+0000 7f894c200700 1 -- 172.21.15.44:0/20065690 >> 172.21.15.44:0/20065690 conn(0x7f89440f8540 msgr2=0x7f89440fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.786+0000 7f894c200700 1 -- 172.21.15.44:0/20065690 shutdown_connections 2024-04-16T11:49:40.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.786+0000 7f894c200700 1 -- 172.21.15.44:0/20065690 wait complete. 2024-04-16T11:49:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.787+0000 7f894c200700 1 Processor -- start 2024-04-16T11:49:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.787+0000 7f894c200700 1 -- start start 2024-04-16T11:49:40.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.787+0000 7f894c200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f8944071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.787+0000 7f894c200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 0x7f8944070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.787+0000 7f894c200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89440708c0 con 0x7f89440fcdc0 2024-04-16T11:49:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.787+0000 7f894c200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8944070a30 con 0x7f89440fd710 2024-04-16T11:49:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8949800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f8944071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8949800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f8944071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53122/0 (socket says 172.21.15.44:53122) 2024-04-16T11:49:40.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8949800700 1 -- 172.21.15.44:0/863095884 learned_addr learned my addr 172.21.15.44:0/863095884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8948e00700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 0x7f8944070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8949800700 1 -- 172.21.15.44:0/863095884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 msgr2=0x7f8944070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8949800700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 0x7f8944070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8949800700 1 -- 172.21.15.44:0/863095884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938005040 con 0x7f89440fcdc0 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.788+0000 7f8948e00700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 0x7f8944070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.789+0000 7f8949800700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f8944071e40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f893800b970 tx=0x7f8938009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.789+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f893800e040 con 0x7f89440fcdc0 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.789+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f893800bbf0 con 0x7f89440fcdc0 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.789+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8944070cb0 con 0x7f89440fcdc0 2024-04-16T11:49:40.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.789+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f893801c630 con 0x7f89440fcdc0 2024-04-16T11:49:40.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.789+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8944115ad0 con 0x7f89440fcdc0 2024-04-16T11:49:40.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.791+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894405f5f0 con 0x7f89440fcdc0 2024-04-16T11:49:40.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.791+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f893800bd60 con 0x7f89440fcdc0 2024-04-16T11:49:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.792+0000 7f8941600700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f892c06ec10 0x7f892c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.792+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f893808fdd0 con 0x7f89440fcdc0 2024-04-16T11:49:40.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.793+0000 7f8948e00700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f892c06ec10 0x7f892c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:40.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.793+0000 7f8948e00700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f892c06ec10 0x7f892c0710c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8930009fd0 tx=0x7f893000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:40.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:40.798+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8938058a60 con 0x7f89440fcdc0 2024-04-16T11:49:41.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.040+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8944102430 con 0x7f892c06ec10 2024-04-16T11:49:41.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.057+0000 7f8941600700 1 -- 172.21.15.44:0/863095884 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f8944102430 con 0x7f892c06ec10 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (6m) 107s ago 17m 13.2M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (3m) 107s ago 17m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (3m) 8s ago 16m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (6m) 107s ago 16m 74.1M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (10m) 107s ago 10m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (10m) 107s ago 10m 23.5M - 17.2.7-847-g45317f95 b333581ccdf5 d011c752c082 2024-04-16T11:49:41.054 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (9m) 8s ago 9m 14.1M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (10m) 8s ago 10m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (6m) 107s ago 19m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (5m) 8s ago 15m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (5m) 107s ago 19m 46.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (4m) 8s ago 15m 36.1M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (6m) 107s ago 17m 7880k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (6m) 8s ago 16m 7746k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (2m) 107s ago 14m 50.7M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (2m) 107s ago 14m 42.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (115s) 107s ago 13m 11.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (85s) 8s ago 13m 42.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (56s) 8s ago 12m 41.8M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (13s) 8s ago 11m 11.1M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:49:41.055 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (5m) 107s ago 16m 56.6M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.059+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f892c06ec10 msgr2=0x7f892c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.059+0000 7f894c200700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f892c06ec10 0x7f892c0710c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8930009fd0 tx=0x7f893000b040 comp rx=0 tx=0).stop 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 msgr2=0x7f8944071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f8944071e40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f893800b970 tx=0x7f8938009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 shutdown_connections 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f892c06ec10 0x7f892c0710c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89440fcdc0 0x7f8944071e40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 --2- 172.21.15.44:0/863095884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89440fd710 0x7f8944070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 >> 172.21.15.44:0/863095884 conn(0x7f89440f8540 msgr2=0x7f89440f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 shutdown_connections 2024-04-16T11:49:41.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.060+0000 7f894c200700 1 -- 172.21.15.44:0/863095884 wait complete. 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 -- 172.21.15.44:0/4258824478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 msgr2=0x7f37bc104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 --2- 172.21.15.44:0/4258824478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc104b40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f37a8004940 tx=0x7f37a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 -- 172.21.15.44:0/4258824478 shutdown_connections 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 --2- 172.21.15.44:0/4258824478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc104b40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 --2- 172.21.15.44:0/4258824478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37bc103420 0x7f37bc103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 -- 172.21.15.44:0/4258824478 >> 172.21.15.44:0/4258824478 conn(0x7f37bc072640 msgr2=0x7f37bc074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:41.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 -- 172.21.15.44:0/4258824478 shutdown_connections 2024-04-16T11:49:41.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.220+0000 7f37c3000700 1 -- 172.21.15.44:0/4258824478 wait complete. 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.221+0000 7f37c3000700 1 Processor -- start 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.221+0000 7f37c3000700 1 -- start start 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.221+0000 7f37c3000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37bc103420 0x7f37bc113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.221+0000 7f37c3000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.221+0000 7f37c3000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc114a90 con 0x7f37bc1046d0 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.221+0000 7f37c3000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc114c00 con 0x7f37bc103420 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53144/0 (socket says 172.21.15.44:53144) 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bb400700 1 -- 172.21.15.44:0/3349270590 learned_addr learned my addr 172.21.15.44:0/3349270590 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bbe00700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37bc103420 0x7f37bc113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bb400700 1 -- 172.21.15.44:0/3349270590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37bc103420 msgr2=0x7f37bc113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bb400700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37bc103420 0x7f37bc113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.222+0000 7f37bb400700 1 -- 172.21.15.44:0/3349270590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a8005040 con 0x7f37bc1046d0 2024-04-16T11:49:41.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.223+0000 7f37bb400700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc114490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009150 tx=0x7f37a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:41.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.223+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37a800e040 con 0x7f37bc1046d0 2024-04-16T11:49:41.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.223+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37a800bcc0 con 0x7f37bc1046d0 2024-04-16T11:49:41.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.223+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc119600 con 0x7f37bc1046d0 2024-04-16T11:49:41.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.223+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37a801c630 con 0x7f37bc1046d0 2024-04-16T11:49:41.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.223+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc119bc0 con 0x7f37bc1046d0 2024-04-16T11:49:41.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.224+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc04c710 con 0x7f37bc1046d0 2024-04-16T11:49:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.225+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f37a8013420 con 0x7f37bc1046d0 2024-04-16T11:49:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.227+0000 7f37b3e00700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f37a006ec00 0x7f37a00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.227+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f37a808f630 con 0x7f37bc1046d0 2024-04-16T11:49:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.227+0000 7f37bbe00700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f37a006ec00 0x7f37a00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.228+0000 7f37bbe00700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f37a006ec00 0x7f37a00710b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f37ac004990 tx=0x7f37ac009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:41.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.232+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f37a805be90 con 0x7f37bc1046d0 2024-04-16T11:49:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:49:41.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: pgmap v191: 65 pgs: 14 peering, 2 active+undersized, 49 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all osd 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.478 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T11:49:41.479 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:41 smithi044 ceph-mon[93020]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:41.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.569+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f37bc119f70 con 0x7f37bc1046d0 2024-04-16T11:49:41.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.571+0000 7f37b3e00700 1 -- 172.21.15.44:0/3349270590 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+733 (secure 0 0 0) 0x7f37a805ba20 con 0x7f37bc1046d0 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:49:41.569 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4, 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 10 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:49:41.570 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:49:41.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f37a006ec00 msgr2=0x7f37a00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f37a006ec00 0x7f37a00710b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f37ac004990 tx=0x7f37ac009ee0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 msgr2=0x7f37bc114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc114490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009150 tx=0x7f37a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 shutdown_connections 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f37a006ec00 0x7f37a00710b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37bc103420 0x7f37bc113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 --2- 172.21.15.44:0/3349270590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37bc1046d0 0x7f37bc114490 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009150 tx=0x7f37a8009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 >> 172.21.15.44:0/3349270590 conn(0x7f37bc072640 msgr2=0x7f37bc073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 shutdown_connections 2024-04-16T11:49:41.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.575+0000 7f37c3000700 1 -- 172.21.15.44:0/3349270590 wait complete. 2024-04-16T11:49:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: pgmap v191: 65 pgs: 14 peering, 2 active+undersized, 49 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all osd 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:41.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T11:49:41.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:41 smithi045 ceph-mon[81423]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.726+0000 7f772fc00700 1 -- 172.21.15.44:0/1574414603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 msgr2=0x7f7728070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.726+0000 7f772fc00700 1 --2- 172.21.15.44:0/1574414603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f7728070e10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7714004940 tx=0x7f7714009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.727+0000 7f772fc00700 1 -- 172.21.15.44:0/1574414603 shutdown_connections 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.727+0000 7f772fc00700 1 --2- 172.21.15.44:0/1574414603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f7728070e10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.727+0000 7f772fc00700 1 --2- 172.21.15.44:0/1574414603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7728071f50 0x7f7728070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.727+0000 7f772fc00700 1 -- 172.21.15.44:0/1574414603 >> 172.21.15.44:0/1574414603 conn(0x7f77280f8530 msgr2=0x7f77280fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.727+0000 7f772fc00700 1 -- 172.21.15.44:0/1574414603 shutdown_connections 2024-04-16T11:49:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.727+0000 7f772fc00700 1 -- 172.21.15.44:0/1574414603 wait complete. 2024-04-16T11:49:41.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.728+0000 7f772fc00700 1 Processor -- start 2024-04-16T11:49:41.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.728+0000 7f772fc00700 1 -- start start 2024-04-16T11:49:41.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.728+0000 7f772fc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f772810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:41.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.728+0000 7f772fc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7728071f50 0x7f7728110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:41.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.728+0000 7f772fc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7728110820 con 0x7f77280709a0 2024-04-16T11:49:41.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.728+0000 7f772fc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7728111010 con 0x7f7728071f50 2024-04-16T11:49:41.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.729+0000 7f772d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f772810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:41.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.729+0000 7f772d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f772810fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49638/0 (socket says 172.21.15.44:49638) 2024-04-16T11:49:41.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.729+0000 7f772d200700 1 -- 172.21.15.44:0/3255913092 learned_addr learned my addr 172.21.15.44:0/3255913092 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.729+0000 7f772d200700 1 -- 172.21.15.44:0/3255913092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7728071f50 msgr2=0x7f7728110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.729+0000 7f772d200700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7728071f50 0x7f7728110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.729+0000 7f772d200700 1 -- 172.21.15.44:0/3255913092 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714005040 con 0x7f77280709a0 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.730+0000 7f772d200700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f772810fc50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f771c00b790 tx=0x7f771c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.730+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f771c00f6f0 con 0x7f77280709a0 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.730+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f771c00fd20 con 0x7f77280709a0 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.730+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77281112f0 con 0x7f77280709a0 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.730+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f771c0103a0 con 0x7f77280709a0 2024-04-16T11:49:41.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.730+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77281118b0 con 0x7f77280709a0 2024-04-16T11:49:41.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.731+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77281095c0 con 0x7f77280709a0 2024-04-16T11:49:41.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.732+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f771c011400 con 0x7f77280709a0 2024-04-16T11:49:41.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.733+0000 7f7724c00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f771006ec00 0x7f77100710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:41.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.734+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f771c08e770 con 0x7f77280709a0 2024-04-16T11:49:41.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.734+0000 7f7727e00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f771006ec00 0x7f77100710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:41.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.735+0000 7f7727e00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f771006ec00 0x7f77100710b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7714004f90 tx=0x7f7714009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:41.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:41.739+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f771c05afd0 con 0x7f77280709a0 2024-04-16T11:49:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.014+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7728111c90 con 0x7f77280709a0 2024-04-16T11:49:42.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.015+0000 7f7724c00700 1 -- 172.21.15.44:0/3255913092 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f771c05ab60 con 0x7f77280709a0 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:e13 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 12 2024-04-16T11:49:42.012 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:39:47.009160+0000 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:49:42.013 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14456} 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:14456} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.44:6826/2456403196,v1:172.21.15.44:6827/2456403196] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:49:42.014 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:49:42.015 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:14470} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:42.015 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:42.015 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:49:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.017+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f771006ec00 msgr2=0x7f77100710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.017+0000 7f772fc00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f771006ec00 0x7f77100710b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7714004f90 tx=0x7f7714009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:42.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.017+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 msgr2=0x7f772810fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.017+0000 7f772fc00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f772810fc50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f771c00b790 tx=0x7f771c009220 comp rx=0 tx=0).stop 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 shutdown_connections 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f771006ec00 0x7f77100710b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f77280709a0 0x7f772810fc50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 --2- 172.21.15.44:0/3255913092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7728071f50 0x7f7728110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 >> 172.21.15.44:0/3255913092 conn(0x7f77280f8530 msgr2=0x7f7728102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 shutdown_connections 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.018+0000 7f772fc00700 1 -- 172.21.15.44:0/3255913092 wait complete. 2024-04-16T11:49:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.176+0000 7ff98cc00700 1 -- 172.21.15.44:0/1471939122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 msgr2=0x7ff9880ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.176+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1471939122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff9880ffa30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff978004940 tx=0x7ff978009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.176+0000 7ff98cc00700 1 -- 172.21.15.44:0/1471939122 shutdown_connections 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.176+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1471939122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9880fff70 0x7ff988102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.176+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1471939122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff9880ffa30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.176+0000 7ff98cc00700 1 -- 172.21.15.44:0/1471939122 >> 172.21.15.44:0/1471939122 conn(0x7ff9880f74c0 msgr2=0x7ff9880f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.177+0000 7ff98cc00700 1 -- 172.21.15.44:0/1471939122 shutdown_connections 2024-04-16T11:49:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.177+0000 7ff98cc00700 1 -- 172.21.15.44:0/1471939122 wait complete. 2024-04-16T11:49:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.178+0000 7ff98cc00700 1 Processor -- start 2024-04-16T11:49:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.178+0000 7ff98cc00700 1 -- start start 2024-04-16T11:49:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.178+0000 7ff98cc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff98810da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff98cc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9880fff70 0x7ff98810dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff98cc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98810e660 con 0x7ff9880662f0 2024-04-16T11:49:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff98cc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98810ee00 con 0x7ff9880fff70 2024-04-16T11:49:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff98810da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff98810da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49646/0 (socket says 172.21.15.44:49646) 2024-04-16T11:49:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985e00700 1 -- 172.21.15.44:0/1615461146 learned_addr learned my addr 172.21.15.44:0/1615461146 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985400700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9880fff70 0x7ff98810dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985e00700 1 -- 172.21.15.44:0/1615461146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9880fff70 msgr2=0x7ff98810dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985e00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9880fff70 0x7ff98810dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.179+0000 7ff985e00700 1 -- 172.21.15.44:0/1615461146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978005040 con 0x7ff9880662f0 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.180+0000 7ff985e00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff98810da90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b3b0 tx=0x7ff978009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.180+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97800e040 con 0x7ff9880662f0 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.180+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff97800bbf0 con 0x7ff9880662f0 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.180+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98810f080 con 0x7ff9880662f0 2024-04-16T11:49:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.180+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97801c630 con 0x7ff9880662f0 2024-04-16T11:49:42.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.180+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98810f640 con 0x7ff9880662f0 2024-04-16T11:49:42.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.182+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98805f5f0 con 0x7ff9880662f0 2024-04-16T11:49:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.182+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff978013420 con 0x7ff9880662f0 2024-04-16T11:49:42.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.183+0000 7ff97e000700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96c06ec10 0x7ff96c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:42.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.183+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6745+0+0 (secure 0 0 0) 0x7ff97808fc30 con 0x7ff9880662f0 2024-04-16T11:49:42.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.184+0000 7ff985400700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96c06ec10 0x7ff96c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:42.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.184+0000 7ff985400700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96c06ec10 0x7ff96c0710c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff974006fd0 tx=0x7ff974008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:42.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.188+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff97805c530 con 0x7ff9880662f0 2024-04-16T11:49:42.403 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:42.403 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T11:49:42.403 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:49:42.403 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T11:49:42.403 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: osdmap e87: 6 total, 6 up, 6 in 2024-04-16T11:49:42.404 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3349270590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:42.404 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi044.ufhhlc"]}]: dispatch 2024-04-16T11:49:42.404 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: Upgrade: It appears safe to stop mds.cephfs.smithi044.ufhhlc 2024-04-16T11:49:42.404 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:42 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3255913092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:49:42.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.438+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff98810f920 con 0x7ff96c06ec10 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.440+0000 7ff97e000700 1 -- 172.21.15.44:0/1615461146 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff98810f920 con 0x7ff96c06ec10 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:49:42.437 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "12/21 daemons upgraded", 2024-04-16T11:49:42.438 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T11:49:42.438 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:49:42.438 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:49:42.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96c06ec10 msgr2=0x7ff96c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96c06ec10 0x7ff96c0710c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff974006fd0 tx=0x7ff974008040 comp rx=0 tx=0).stop 2024-04-16T11:49:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 msgr2=0x7ff98810da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff98810da90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b3b0 tx=0x7ff978009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 shutdown_connections 2024-04-16T11:49:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96c06ec10 0x7ff96c0710c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9880662f0 0x7ff98810da90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 --2- 172.21.15.44:0/1615461146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9880fff70 0x7ff98810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.443+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 >> 172.21.15.44:0/1615461146 conn(0x7ff9880f74c0 msgr2=0x7ff9880f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:42.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.444+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 shutdown_connections 2024-04-16T11:49:42.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.444+0000 7ff98cc00700 1 -- 172.21.15.44:0/1615461146 wait complete. 2024-04-16T11:49:42.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:42.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: osdmap e87: 6 total, 6 up, 6 in 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3349270590' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi044.ufhhlc"]}]: dispatch 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: Upgrade: It appears safe to stop mds.cephfs.smithi044.ufhhlc 2024-04-16T11:49:42.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:42 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3255913092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:49:42.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 -- 172.21.15.44:0/3824720513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 msgr2=0x7fde240fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 --2- 172.21.15.44:0/3824720513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde240fd250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde18004940 tx=0x7fde18009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:42.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 -- 172.21.15.44:0/3824720513 shutdown_connections 2024-04-16T11:49:42.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 --2- 172.21.15.44:0/3824720513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde240fe040 0x7fde240fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 --2- 172.21.15.44:0/3824720513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde240fd250 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 -- 172.21.15.44:0/3824720513 >> 172.21.15.44:0/3824720513 conn(0x7fde240f8530 msgr2=0x7fde240fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:42.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 -- 172.21.15.44:0/3824720513 shutdown_connections 2024-04-16T11:49:42.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.593+0000 7fde2c200700 1 -- 172.21.15.44:0/3824720513 wait complete. 2024-04-16T11:49:42.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.594+0000 7fde2c200700 1 Processor -- start 2024-04-16T11:49:42.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.594+0000 7fde2c200700 1 -- start start 2024-04-16T11:49:42.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde2c200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde24114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:42.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde2c200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde240fe040 0x7fde241145c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:42.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde2c200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde24114bc0 con 0x7fde240fce40 2024-04-16T11:49:42.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde2c200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde24115400 con 0x7fde240fe040 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde29800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde24114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde29800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde24114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49674/0 (socket says 172.21.15.44:49674) 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde29800700 1 -- 172.21.15.44:0/3192181180 learned_addr learned my addr 172.21.15.44:0/3192181180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.595+0000 7fde28e00700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde240fe040 0x7fde241145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.596+0000 7fde29800700 1 -- 172.21.15.44:0/3192181180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde240fe040 msgr2=0x7fde241145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.596+0000 7fde29800700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde240fe040 0x7fde241145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.596+0000 7fde29800700 1 -- 172.21.15.44:0/3192181180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde18005040 con 0x7fde240fce40 2024-04-16T11:49:42.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.596+0000 7fde29800700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde24114080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde1800b970 tx=0x7fde18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:42.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.596+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde1800e040 con 0x7fde240fce40 2024-04-16T11:49:42.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.597+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde24115680 con 0x7fde240fce40 2024-04-16T11:49:42.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.597+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde1800bbf0 con 0x7fde240fce40 2024-04-16T11:49:42.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.597+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde1801c630 con 0x7fde240fce40 2024-04-16T11:49:42.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.597+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde24115bc0 con 0x7fde240fce40 2024-04-16T11:49:42.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.598+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde241019d0 con 0x7fde240fce40 2024-04-16T11:49:42.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.599+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fde1800bd60 con 0x7fde240fce40 2024-04-16T11:49:42.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.600+0000 7fde21600700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fde0c06ec10 0x7fde0c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:49:42.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.600+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fde1808fc60 con 0x7fde240fce40 2024-04-16T11:49:42.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.600+0000 7fde28e00700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fde0c06ec10 0x7fde0c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:49:42.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.601+0000 7fde28e00700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fde0c06ec10 0x7fde0c0710c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fde10007fd0 tx=0x7fde10009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:49:42.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.607+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fde1805c560 con 0x7fde240fce40 2024-04-16T11:49:42.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.928+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fde24105a60 con 0x7fde240fce40 2024-04-16T11:49:42.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.928+0000 7fde21600700 1 -- 172.21.15.44:0/3192181180 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fde24105a60 con 0x7fde240fce40 2024-04-16T11:49:42.925 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fde0c06ec10 msgr2=0x7fde0c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fde0c06ec10 0x7fde0c0710c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fde10007fd0 tx=0x7fde10009040 comp rx=0 tx=0).stop 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 msgr2=0x7fde24114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde24114080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde1800b970 tx=0x7fde18009e30 comp rx=0 tx=0).stop 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 shutdown_connections 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fde0c06ec10 0x7fde0c0710c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde240fce40 0x7fde24114080 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 --2- 172.21.15.44:0/3192181180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde240fe040 0x7fde241145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 >> 172.21.15.44:0/3192181180 conn(0x7fde240f8530 msgr2=0x7fde240f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 shutdown_connections 2024-04-16T11:49:42.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:49:42.931+0000 7fde2c200700 1 -- 172.21.15.44:0/3192181180 wait complete. 2024-04-16T11:49:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:43 smithi044 ceph-mon[93020]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:43 smithi044 ceph-mon[93020]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:43 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3192181180' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:49:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:43 smithi045 ceph-mon[81423]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:43 smithi045 ceph-mon[81423]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:49:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:43 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3192181180' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:49:44.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:44 smithi044 ceph-mon[93020]: Upgrade: Updating mds.cephfs.smithi044.ufhhlc 2024-04-16T11:49:44.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:44 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:44.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:44 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:49:44.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:44 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:49:44.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:44 smithi044 ceph-mon[93020]: Deploying daemon mds.cephfs.smithi044.ufhhlc on smithi044 2024-04-16T11:49:45.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:44 smithi045 ceph-mon[81423]: Upgrade: Updating mds.cephfs.smithi044.ufhhlc 2024-04-16T11:49:45.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:44 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:45.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:44 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.ufhhlc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:49:45.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:44 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:49:45.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:44 smithi045 ceph-mon[81423]: Deploying daemon mds.cephfs.smithi044.ufhhlc on smithi044 2024-04-16T11:49:46.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:45 smithi045 ceph-mon[81423]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:46.090 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:45 smithi044 ceph-mon[93020]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 522, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105500 2024-04-16T11:49:47.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 522, success, outsize: 120 2024-04-16T11:49:47.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 354, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118713 2024-04-16T11:49:47.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 354, success, outsize: 120 2024-04-16T11:49:47.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[93016]: 2024-04-16T11:49:47.142+0000 7f2289600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:49:48.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:48.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:49:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:49:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: osdmap e88: 6 total, 6 up, 6 in 2024-04-16T11:49:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: Standby daemon mds.cephfs.smithi044.hlafhv assigned to filesystem cephfs as rank 0 2024-04-16T11:49:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:49:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:49:48.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:47 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi044.hlafhv=up:replay} 2 up:standby 2024-04-16T11:49:48.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 524, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105501 2024-04-16T11:49:48.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 524, success, outsize: 120 2024-04-16T11:49:48.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118724 2024-04-16T11:49:48.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 356, success, outsize: 120 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: osdmap e88: 6 total, 6 up, 6 in 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: Standby daemon mds.cephfs.smithi044.hlafhv assigned to filesystem cephfs as rank 0 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:49:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:47 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi044.hlafhv=up:replay} 2 up:standby 2024-04-16T11:49:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:49 smithi044 ceph-mon[93020]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00399994 2024-04-16T11:49:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:49 smithi044 ceph-mon[93020]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00499992 2024-04-16T11:49:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:49 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:reconnect 2024-04-16T11:49:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:49 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi044.hlafhv=up:reconnect} 2 up:standby 2024-04-16T11:49:49.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:49 smithi044 ceph-mon[93020]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:49 smithi045 ceph-mon[81423]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00399994 2024-04-16T11:49:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:49 smithi045 ceph-mon[81423]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00499992 2024-04-16T11:49:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:49 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:reconnect 2024-04-16T11:49:49.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:49 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi044.hlafhv=up:reconnect} 2 up:standby 2024-04-16T11:49:49.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:49 smithi045 ceph-mon[81423]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:49:50.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:50 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:rejoin 2024-04-16T11:49:50.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:50 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi044.hlafhv=up:rejoin} 2 up:standby 2024-04-16T11:49:50.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:50 smithi044 ceph-mon[93020]: daemon mds.cephfs.smithi044.hlafhv is now active in filesystem cephfs as rank 0 2024-04-16T11:49:50.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:50 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:rejoin 2024-04-16T11:49:50.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:50 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi044.hlafhv=up:rejoin} 2 up:standby 2024-04-16T11:49:50.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:50 smithi045 ceph-mon[81423]: daemon mds.cephfs.smithi044.hlafhv is now active in filesystem cephfs as rank 0 2024-04-16T11:49:51.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:51 smithi044 ceph-mon[93020]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:49:51.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:51 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:49:51.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:51 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:active 2024-04-16T11:49:51.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:51 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi044.hlafhv=up:active} 2 up:standby 2024-04-16T11:49:51.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:51 smithi044 ceph-mon[93020]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 235 B/s rd, 0 op/s 2024-04-16T11:49:51.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:51 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:51 smithi045 ceph-mon[81423]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:49:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:51 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:49:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:51 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] up:active 2024-04-16T11:49:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:51 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi044.hlafhv=up:active} 2 up:standby 2024-04-16T11:49:51.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:51 smithi045 ceph-mon[81423]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 235 B/s rd, 0 op/s 2024-04-16T11:49:51.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:51 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:53.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:53 smithi045 ceph-mon[81423]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T11:49:53.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:53 smithi044 ceph-mon[93020]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T11:49:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:55 smithi045 ceph-mon[81423]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:49:55.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:55.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:55.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:55 smithi044 ceph-mon[93020]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:49:55.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:49:55.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:49:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:57 smithi044 ceph-mon[93020]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:49:57.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:57 smithi045 ceph-mon[81423]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:49:59.383 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:49:59 smithi044 ceph-mon[93020]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 184 B/s wr, 4 op/s 2024-04-16T11:49:59.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:49:59 smithi045 ceph-mon[81423]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 184 B/s wr, 4 op/s 2024-04-16T11:50:00.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:00.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:00.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:00.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: overall HEALTH_OK 2024-04-16T11:50:00.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:boot 2024-04-16T11:50:00.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi044.hlafhv=up:active} 3 up:standby 2024-04-16T11:50:00.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:50:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: overall HEALTH_OK 2024-04-16T11:50:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:boot 2024-04-16T11:50:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi044.hlafhv=up:active} 3 up:standby 2024-04-16T11:50:01.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:50:01.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:01 smithi044 ceph-mon[93020]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:01 smithi045 ceph-mon[81423]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:03.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:03 smithi044 ceph-mon[93020]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:04.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:03 smithi045 ceph-mon[81423]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:05.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:05 smithi044 ceph-mon[93020]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T11:50:06.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:05 smithi045 ceph-mon[81423]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T11:50:07.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:07 smithi045 ceph-mon[81423]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T11:50:07.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:07.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:07.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:07 smithi044 ceph-mon[93020]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T11:50:07.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:07.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:09.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:09 smithi044 ceph-mon[93020]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:09.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:09.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:10.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:09 smithi045 ceph-mon[81423]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:10.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:10.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:10.988 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:11.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:11.958 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:11 smithi044 ceph-mon[93020]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:12.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:11 smithi045 ceph-mon[81423]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:13.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 -- 172.21.15.44:0/1603828156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 msgr2=0x7ff6b00fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/1603828156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b00fe350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004940 tx=0x7ff69c009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:13.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 -- 172.21.15.44:0/1603828156 shutdown_connections 2024-04-16T11:50:13.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/1603828156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b00fe350 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/1603828156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b00fcc30 0x7ff6b00fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 -- 172.21.15.44:0/1603828156 >> 172.21.15.44:0/1603828156 conn(0x7ff6b00f83d0 msgr2=0x7ff6b00fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 -- 172.21.15.44:0/1603828156 shutdown_connections 2024-04-16T11:50:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.097+0000 7ff6b4a00700 1 -- 172.21.15.44:0/1603828156 wait complete. 2024-04-16T11:50:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.098+0000 7ff6b4a00700 1 Processor -- start 2024-04-16T11:50:13.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.098+0000 7ff6b4a00700 1 -- start start 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6b4a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b00fcc30 0x7ff6b010fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6b4a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b010ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6b4a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01105b0 con 0x7ff6b00fdee0 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6b4a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0110e00 con 0x7ff6b00fcc30 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6aea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b010ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6aea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b010ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50694/0 (socket says 172.21.15.44:50694) 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6aea00700 1 -- 172.21.15.44:0/2203777961 learned_addr learned my addr 172.21.15.44:0/2203777961 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.099+0000 7ff6af400700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b00fcc30 0x7ff6b010fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.100+0000 7ff6aea00700 1 -- 172.21.15.44:0/2203777961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b00fcc30 msgr2=0x7ff6b010fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.100+0000 7ff6aea00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b00fcc30 0x7ff6b010fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.100+0000 7ff6aea00700 1 -- 172.21.15.44:0/2203777961 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c005040 con 0x7ff6b00fdee0 2024-04-16T11:50:13.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.100+0000 7ff6aea00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b010ffb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00b3b0 tx=0x7ff69c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:13.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.101+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69c00e040 con 0x7ff6b00fdee0 2024-04-16T11:50:13.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.101+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b0111080 con 0x7ff6b00fdee0 2024-04-16T11:50:13.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.101+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff69c00bcc0 con 0x7ff6b00fdee0 2024-04-16T11:50:13.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.101+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69c01c630 con 0x7ff6b00fdee0 2024-04-16T11:50:13.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.101+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b01115c0 con 0x7ff6b00fdee0 2024-04-16T11:50:13.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.102+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0101870 con 0x7ff6b00fdee0 2024-04-16T11:50:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.104+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff69c013980 con 0x7ff6b00fdee0 2024-04-16T11:50:13.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.105+0000 7ff6a7400700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff69406ecd0 0x7ff694071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.105+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7ff69c08f6e0 con 0x7ff6b00fdee0 2024-04-16T11:50:13.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.105+0000 7ff6af400700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff69406ecd0 0x7ff694071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.106+0000 7ff6af400700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff69406ecd0 0x7ff694071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0007fd0 tx=0x7ff6a0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:13.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.114+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff69c05be50 con 0x7ff6b00fdee0 2024-04-16T11:50:13.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.367+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6b01058b0 con 0x7ff69406ecd0 2024-04-16T11:50:13.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.370+0000 7ff6a7400700 1 -- 172.21.15.44:0/2203777961 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff6b01058b0 con 0x7ff69406ecd0 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff69406ecd0 msgr2=0x7ff694071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff69406ecd0 0x7ff694071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0007fd0 tx=0x7ff6a0009040 comp rx=0 tx=0).stop 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 msgr2=0x7ff6b010ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b010ffb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00b3b0 tx=0x7ff69c009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 shutdown_connections 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff69406ecd0 0x7ff694071180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6b00fcc30 0x7ff6b010fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 --2- 172.21.15.44:0/2203777961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6b00fdee0 0x7ff6b010ffb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 >> 172.21.15.44:0/2203777961 conn(0x7ff6b00f83d0 msgr2=0x7ff6b00f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:13.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 shutdown_connections 2024-04-16T11:50:13.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.373+0000 7ff6b4a00700 1 -- 172.21.15.44:0/2203777961 wait complete. 2024-04-16T11:50:13.388 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:50:13.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.531+0000 7f4607000700 1 -- 172.21.15.44:0/2143973813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fce40 msgr2=0x7f46000fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.531+0000 7f4607000700 1 --2- 172.21.15.44:0/2143973813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fce40 0x7f46000fd250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f45f0004940 tx=0x7f45f0009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:13.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.531+0000 7f4607000700 1 -- 172.21.15.44:0/2143973813 shutdown_connections 2024-04-16T11:50:13.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.531+0000 7f4607000700 1 --2- 172.21.15.44:0/2143973813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fe040 0x7f46000fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.531+0000 7f4607000700 1 --2- 172.21.15.44:0/2143973813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fce40 0x7f46000fd250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.531+0000 7f4607000700 1 -- 172.21.15.44:0/2143973813 >> 172.21.15.44:0/2143973813 conn(0x7f46000f8530 msgr2=0x7f46000fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:13.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.532+0000 7f4607000700 1 -- 172.21.15.44:0/2143973813 shutdown_connections 2024-04-16T11:50:13.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.532+0000 7f4607000700 1 -- 172.21.15.44:0/2143973813 wait complete. 2024-04-16T11:50:13.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.532+0000 7f4607000700 1 Processor -- start 2024-04-16T11:50:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f4607000700 1 -- start start 2024-04-16T11:50:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f4607000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fce40 0x7f4600071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f4607000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 0x7f4600070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f4607000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46000708c0 con 0x7f46000fe040 2024-04-16T11:50:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f4607000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4600070a30 con 0x7f46000fce40 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f45ff400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 0x7f4600070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f45ff400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 0x7f4600070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50708/0 (socket says 172.21.15.44:50708) 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.533+0000 7f45ff400700 1 -- 172.21.15.44:0/795664643 learned_addr learned my addr 172.21.15.44:0/795664643 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45ffe00700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fce40 0x7f4600071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45ff400700 1 -- 172.21.15.44:0/795664643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fce40 msgr2=0x7f4600071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45ff400700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fce40 0x7f4600071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45ff400700 1 -- 172.21.15.44:0/795664643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0005040 con 0x7f46000fe040 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45ffe00700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fce40 0x7f4600071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:50:13.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45ff400700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 0x7f4600070310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45ec002a80 tx=0x7f45ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:13.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.534+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45ec007cc0 con 0x7f46000fe040 2024-04-16T11:50:13.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.535+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4600070d10 con 0x7f46000fe040 2024-04-16T11:50:13.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.535+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45ec00f040 con 0x7f46000fe040 2024-04-16T11:50:13.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.535+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45ec0096f0 con 0x7f46000fe040 2024-04-16T11:50:13.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.535+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4600115a30 con 0x7f46000fe040 2024-04-16T11:50:13.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.536+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460005f5f0 con 0x7f46000fe040 2024-04-16T11:50:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.538+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f45ec00d690 con 0x7f46000fe040 2024-04-16T11:50:13.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.539+0000 7f45f7e00700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f45e406ec00 0x7f45e40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.539+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f45ec08e190 con 0x7f46000fe040 2024-04-16T11:50:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.539+0000 7f45ffe00700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f45e406ec00 0x7f45e40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.540+0000 7f45ffe00700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f45e406ec00 0x7f45e40710b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f45f000b970 tx=0x7f45f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:13.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.544+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f45ec056dd0 con 0x7f46000fe040 2024-04-16T11:50:13.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 358, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120024 2024-04-16T11:50:13.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 358, success, outsize: 120 2024-04-16T11:50:13.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.801+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4600102360 con 0x7f45e406ec00 2024-04-16T11:50:13.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.803+0000 7f45f7e00700 1 -- 172.21.15.44:0/795664643 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4600102360 con 0x7f45e406ec00 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f45e406ec00 msgr2=0x7f45e40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f45e406ec00 0x7f45e40710b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f45f000b970 tx=0x7f45f0009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 msgr2=0x7f4600070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 0x7f4600070310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45ec002a80 tx=0x7f45ec0046b0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 shutdown_connections 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f45e406ec00 0x7f45e40710b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46000fce40 0x7f4600071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 --2- 172.21.15.44:0/795664643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46000fe040 0x7f4600070310 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 >> 172.21.15.44:0/795664643 conn(0x7f46000f8530 msgr2=0x7f46000f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:13.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 shutdown_connections 2024-04-16T11:50:13.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.806+0000 7f4607000700 1 -- 172.21.15.44:0/795664643 wait complete. 2024-04-16T11:50:13.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 360, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120108 2024-04-16T11:50:13.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 360, success, outsize: 120 2024-04-16T11:50:13.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 362, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120194 2024-04-16T11:50:13.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 362, success, outsize: 120 2024-04-16T11:50:13.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120285 2024-04-16T11:50:13.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 364, success, outsize: 120 2024-04-16T11:50:13.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 366, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120373 2024-04-16T11:50:13.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 366, success, outsize: 120 2024-04-16T11:50:13.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 368, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120455 2024-04-16T11:50:13.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 368, success, outsize: 120 2024-04-16T11:50:13.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 370, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120540 2024-04-16T11:50:13.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 370, success, outsize: 120 2024-04-16T11:50:13.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 372, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120631 2024-04-16T11:50:13.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 372, success, outsize: 120 2024-04-16T11:50:13.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 374, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120719 2024-04-16T11:50:13.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 374, success, outsize: 120 2024-04-16T11:50:13.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 376, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120802 2024-04-16T11:50:13.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 376, success, outsize: 120 2024-04-16T11:50:13.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 378, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120897 2024-04-16T11:50:13.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 378, success, outsize: 120 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.976+0000 7fded6600700 1 -- 172.21.15.44:0/3146950253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 msgr2=0x7fded0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 --2- 172.21.15.44:0/3146950253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0070460 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdec0004940 tx=0x7fdec0009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 -- 172.21.15.44:0/3146950253 shutdown_connections 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 --2- 172.21.15.44:0/3146950253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded00709a0 0x7fded0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 --2- 172.21.15.44:0/3146950253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0070460 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 -- 172.21.15.44:0/3146950253 >> 172.21.15.44:0/3146950253 conn(0x7fded00f8530 msgr2=0x7fded00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 -- 172.21.15.44:0/3146950253 shutdown_connections 2024-04-16T11:50:13.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.977+0000 7fded6600700 1 -- 172.21.15.44:0/3146950253 wait complete. 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fded6600700 1 Processor -- start 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fded6600700 1 -- start start 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fded6600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded00709a0 0x7fded010fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fded6600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fded6600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded0110870 con 0x7fded0071f50 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fded6600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded0110fb0 con 0x7fded00709a0 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fdecea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.978+0000 7fdecea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50720/0 (socket says 172.21.15.44:50720) 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.979+0000 7fdecea00700 1 -- 172.21.15.44:0/1107514224 learned_addr learned my addr 172.21.15.44:0/1107514224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.979+0000 7fdecf400700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded00709a0 0x7fded010fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.979+0000 7fdecea00700 1 -- 172.21.15.44:0/1107514224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded00709a0 msgr2=0x7fded010fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.979+0000 7fdecea00700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded00709a0 0x7fded010fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.979+0000 7fdecea00700 1 -- 172.21.15.44:0/1107514224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0005040 con 0x7fded0071f50 2024-04-16T11:50:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.979+0000 7fdecea00700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0110270 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdebc002a50 tx=0x7fdebc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:13.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.980+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdebc007cc0 con 0x7fded0071f50 2024-04-16T11:50:13.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.980+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded0111290 con 0x7fded0071f50 2024-04-16T11:50:13.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.980+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded0111850 con 0x7fded0071f50 2024-04-16T11:50:13.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.980+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdebc00f040 con 0x7fded0071f50 2024-04-16T11:50:13.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.980+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdebc009510 con 0x7fded0071f50 2024-04-16T11:50:13.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.981+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb00052e0 con 0x7fded0071f50 2024-04-16T11:50:13.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.982+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdebc009670 con 0x7fded0071f50 2024-04-16T11:50:13.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.983+0000 7fdec7400700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdeb406ec10 0x7fdeb40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:13.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.984+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fdebc08e740 con 0x7fded0071f50 2024-04-16T11:50:13.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.984+0000 7fdecf400700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdeb406ec10 0x7fdeb40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:13.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.984+0000 7fdecf400700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdeb406ec10 0x7fdeb40710c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009fd0 tx=0x7fdec0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:13.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:13.989+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdebc05af60 con 0x7fded0071f50 2024-04-16T11:50:13.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 380, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120997 2024-04-16T11:50:13.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 380, success, outsize: 120 2024-04-16T11:50:14.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 382, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121085 2024-04-16T11:50:14.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 382, success, outsize: 120 2024-04-16T11:50:14.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 384, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121167 2024-04-16T11:50:14.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 384, success, outsize: 120 2024-04-16T11:50:14.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 386, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121252 2024-04-16T11:50:14.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 386, success, outsize: 120 2024-04-16T11:50:14.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 388, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121343 2024-04-16T11:50:14.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 388, success, outsize: 120 2024-04-16T11:50:14.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 390, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121431 2024-04-16T11:50:14.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 390, success, outsize: 120 2024-04-16T11:50:14.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 392, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121513 2024-04-16T11:50:14.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 392, success, outsize: 120 2024-04-16T11:50:14.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:13 smithi045 ceph-mon[81423]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:14.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 394, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121598 2024-04-16T11:50:14.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 394, success, outsize: 120 2024-04-16T11:50:14.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 396, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121689 2024-04-16T11:50:14.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 396, success, outsize: 120 2024-04-16T11:50:14.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 398, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121777 2024-04-16T11:50:14.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 398, success, outsize: 120 2024-04-16T11:50:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 400, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121859 2024-04-16T11:50:14.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 400, success, outsize: 120 2024-04-16T11:50:14.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 402, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121944 2024-04-16T11:50:14.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 402, success, outsize: 120 2024-04-16T11:50:14.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 404, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122035 2024-04-16T11:50:14.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 404, success, outsize: 120 2024-04-16T11:50:14.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 406, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122123 2024-04-16T11:50:14.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 406, success, outsize: 120 2024-04-16T11:50:14.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 408, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122205 2024-04-16T11:50:14.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 408, success, outsize: 120 2024-04-16T11:50:14.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 410, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122290 2024-04-16T11:50:14.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 410, success, outsize: 120 2024-04-16T11:50:14.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:13 smithi044 ceph-mon[93020]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:14.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 412, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122381 2024-04-16T11:50:14.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 412, success, outsize: 120 2024-04-16T11:50:14.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.239+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeb0000bf0 con 0x7fdeb406ec10 2024-04-16T11:50:14.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.259+0000 7fdec7400700 1 -- 172.21.15.44:0/1107514224 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fdeb0000bf0 con 0x7fdeb406ec10 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (7m) 6s ago 17m 13.6M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (4m) 6s ago 18m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (4m) 41s ago 16m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (7m) 6s ago 17m 74.2M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (10m) 6s ago 10m 17.3M - 17.2.7-847-g45317f95 b333581ccdf5 1be13b1b7cee 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (15s) 6s ago 10m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (10m) 41s ago 10m 14.1M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (10m) 41s ago 10m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (6m) 6s ago 19m 508M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (6m) 41s ago 16m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (5m) 6s ago 19m 49.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:50:14.257 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (5m) 41s ago 16m 36.1M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (7m) 6s ago 18m 8082k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (7m) 41s ago 16m 7746k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (3m) 6s ago 15m 53.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (2m) 6s ago 14m 44.5M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (2m) 6s ago 14m 43.8M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (118s) 41s ago 13m 42.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (89s) 41s ago 13m 41.8M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (46s) 41s ago 12m 11.1M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:50:14.258 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (6m) 6s ago 17m 57.7M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:50:14.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdeb406ec10 msgr2=0x7fdeb40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdeb406ec10 0x7fdeb40710c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009fd0 tx=0x7fdec0009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:14.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 msgr2=0x7fded0110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0110270 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdebc002a50 tx=0x7fdebc0046b0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 shutdown_connections 2024-04-16T11:50:14.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdeb406ec10 0x7fdeb40710c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded00709a0 0x7fded010fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 --2- 172.21.15.44:0/1107514224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fded0071f50 0x7fded0110270 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 >> 172.21.15.44:0/1107514224 conn(0x7fded00f8530 msgr2=0x7fded00f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:14.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 shutdown_connections 2024-04-16T11:50:14.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.263+0000 7fded6600700 1 -- 172.21.15.44:0/1107514224 wait complete. 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 -- 172.21.15.44:0/2379926239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 msgr2=0x7f0bb40ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2379926239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb40ff410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8004940 tx=0x7f0ba8009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 -- 172.21.15.44:0/2379926239 shutdown_connections 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2379926239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb41002b0 0x7f0bb4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2379926239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb40ff410 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 -- 172.21.15.44:0/2379926239 >> 172.21.15.44:0/2379926239 conn(0x7f0bb40fa950 msgr2=0x7f0bb40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 -- 172.21.15.44:0/2379926239 shutdown_connections 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.421+0000 7f0bbc200700 1 -- 172.21.15.44:0/2379926239 wait complete. 2024-04-16T11:50:14.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.422+0000 7f0bbc200700 1 Processor -- start 2024-04-16T11:50:14.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.422+0000 7f0bbc200700 1 -- start start 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bbc200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb410fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bbc200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb41002b0 0x7f0bb4110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bbc200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4110760 con 0x7f0bb40ff000 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bbc200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4110fb0 con 0x7f0bb41002b0 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb410fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bb9800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb410fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50730/0 (socket says 172.21.15.44:50730) 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bb9800700 1 -- 172.21.15.44:0/2761511118 learned_addr learned my addr 172.21.15.44:0/2761511118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.423+0000 7f0bb8e00700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb41002b0 0x7f0bb4110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb9800700 1 -- 172.21.15.44:0/2761511118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb41002b0 msgr2=0x7f0bb4110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb9800700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb41002b0 0x7f0bb4110160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb9800700 1 -- 172.21.15.44:0/2761511118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8005040 con 0x7f0bb40ff000 2024-04-16T11:50:14.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb9800700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb410fc20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009150 tx=0x7f0ba8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:14.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ba800e040 con 0x7f0bb40ff000 2024-04-16T11:50:14.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ba800bbf0 con 0x7f0bb40ff000 2024-04-16T11:50:14.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb4111230 con 0x7f0bb40ff000 2024-04-16T11:50:14.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.424+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ba801c630 con 0x7f0bb40ff000 2024-04-16T11:50:14.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.425+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb4111770 con 0x7f0bb40ff000 2024-04-16T11:50:14.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.426+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb405f5f0 con 0x7f0bb40ff000 2024-04-16T11:50:14.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.427+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0ba800bd60 con 0x7f0bb40ff000 2024-04-16T11:50:14.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.428+0000 7f0bb1600700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b9c06ec10 0x7f0b9c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:14.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.428+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f0ba808fd80 con 0x7f0bb40ff000 2024-04-16T11:50:14.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.428+0000 7f0bb8e00700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b9c06ec10 0x7f0b9c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:14.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.429+0000 7f0bb8e00700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b9c06ec10 0x7f0b9c0710c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0004990 tx=0x7f0ba0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.433+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0ba805c590 con 0x7f0bb40ff000 2024-04-16T11:50:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.778+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0bb4111af0 con 0x7f0bb40ff000 2024-04-16T11:50:14.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.780+0000 7f0bb1600700 1 -- 172.21.15.44:0/2761511118 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f0ba805c120 con 0x7f0bb40ff000 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T11:50:14.778 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T11:50:14.779 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:14.779 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:50:14.779 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T11:50:14.779 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 11 2024-04-16T11:50:14.779 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:50:14.779 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:50:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.782+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b9c06ec10 msgr2=0x7f0b9c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.782+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b9c06ec10 0x7f0b9c0710c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0004990 tx=0x7f0ba0009ee0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.782+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 msgr2=0x7f0bb410fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.782+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb410fc20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009150 tx=0x7f0ba8009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.782+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 shutdown_connections 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.783+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b9c06ec10 0x7f0b9c0710c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.783+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0bb40ff000 0x7f0bb410fc20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.783+0000 7f0bbc200700 1 --2- 172.21.15.44:0/2761511118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bb41002b0 0x7f0bb4110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.783+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 >> 172.21.15.44:0/2761511118 conn(0x7f0bb40fa950 msgr2=0x7f0bb40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.783+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 shutdown_connections 2024-04-16T11:50:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.783+0000 7f0bbc200700 1 -- 172.21.15.44:0/2761511118 wait complete. 2024-04-16T11:50:14.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.937+0000 7f8b2ce00700 1 -- 172.21.15.44:0/3155902521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28070890 msgr2=0x7f8b28070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.937+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/3155902521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28070890 0x7f8b28070d00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8b14004940 tx=0x7f8b14009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:14.935 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:14 smithi044 ceph-mon[93020]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:14.935 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:14 smithi044 ceph-mon[93020]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:14.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.937+0000 7f8b2ce00700 1 -- 172.21.15.44:0/3155902521 shutdown_connections 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.937+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/3155902521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28070890 0x7f8b28070d00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.937+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/3155902521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b28071e50 0x7f8b28070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.937+0000 7f8b2ce00700 1 -- 172.21.15.44:0/3155902521 >> 172.21.15.44:0/3155902521 conn(0x7f8b280f8540 msgr2=0x7f8b280fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.938+0000 7f8b2ce00700 1 -- 172.21.15.44:0/3155902521 shutdown_connections 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.938+0000 7f8b2ce00700 1 -- 172.21.15.44:0/3155902521 wait complete. 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.939+0000 7f8b2ce00700 1 Processor -- start 2024-04-16T11:50:14.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.939+0000 7f8b2ce00700 1 -- start start 2024-04-16T11:50:14.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.939+0000 7f8b2ce00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b28070890 0x7f8b2810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:14.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.939+0000 7f8b2ce00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 0x7f8b28110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:14.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.939+0000 7f8b2ce00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b28110790 con 0x7f8b28071e50 2024-04-16T11:50:14.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.939+0000 7f8b2ce00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b28110fe0 con 0x7f8b28070890 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 0x7f8b28110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 0x7f8b28110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50742/0 (socket says 172.21.15.44:50742) 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25400700 1 -- 172.21.15.44:0/422042891 learned_addr learned my addr 172.21.15.44:0/422042891 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25e00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b28070890 0x7f8b2810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25400700 1 -- 172.21.15.44:0/422042891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b28070890 msgr2=0x7f8b2810fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25400700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b28070890 0x7f8b2810fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:14.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.940+0000 7f8b25400700 1 -- 172.21.15.44:0/422042891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b14005040 con 0x7f8b28071e50 2024-04-16T11:50:14.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.941+0000 7f8b25400700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 0x7f8b28110190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400b3b0 tx=0x7f8b14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:14.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.941+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b1400e040 con 0x7f8b28071e50 2024-04-16T11:50:14.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.941+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b14004290 con 0x7f8b28071e50 2024-04-16T11:50:14.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.941+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b1401c630 con 0x7f8b28071e50 2024-04-16T11:50:14.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.941+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b28111260 con 0x7f8b28071e50 2024-04-16T11:50:14.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.941+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b281117a0 con 0x7f8b28071e50 2024-04-16T11:50:14.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.943+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b2805f5f0 con 0x7f8b28071e50 2024-04-16T11:50:14.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.943+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8b1400bbf0 con 0x7f8b28071e50 2024-04-16T11:50:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.945+0000 7f8b1e000700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8b0c06ec10 0x7f8b0c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.945+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f8b1408fde0 con 0x7f8b28071e50 2024-04-16T11:50:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.945+0000 7f8b25e00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8b0c06ec10 0x7f8b0c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.946+0000 7f8b25e00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8b0c06ec10 0x7f8b0c0710c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009fd0 tx=0x7f8b18009f00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:14.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:14.949+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8b1405c5d0 con 0x7f8b28071e50 2024-04-16T11:50:15.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:14 smithi045 ceph-mon[81423]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:15.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:14 smithi045 ceph-mon[81423]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:15.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.242+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8b28111ad0 con 0x7f8b28071e50 2024-04-16T11:50:15.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.243+0000 7f8b1e000700 1 -- 172.21.15.44:0/422042891 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f8b1405c160 con 0x7f8b28071e50 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:e19 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 18 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:50:15.242 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:49:50.173765+0000 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 88 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=14470} 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:50:15.243 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 14470 members: 14470 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{0:14470} state up:active seq 155 join_fscid=1 addr [v2:172.21.15.44:6828/248184983,v1:172.21.15.44:6829/248184983] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:24277} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:50:15.244 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:50:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8b0c06ec10 msgr2=0x7f8b0c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8b0c06ec10 0x7f8b0c0710c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009fd0 tx=0x7f8b18009f00 comp rx=0 tx=0).stop 2024-04-16T11:50:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 msgr2=0x7f8b28110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 0x7f8b28110190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400b3b0 tx=0x7f8b14009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 shutdown_connections 2024-04-16T11:50:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8b0c06ec10 0x7f8b0c0710c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b28070890 0x7f8b2810fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 --2- 172.21.15.44:0/422042891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b28071e50 0x7f8b28110190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 >> 172.21.15.44:0/422042891 conn(0x7f8b280f8540 msgr2=0x7f8b280f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:15.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 shutdown_connections 2024-04-16T11:50:15.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.247+0000 7f8b2ce00700 1 -- 172.21.15.44:0/422042891 wait complete. 2024-04-16T11:50:15.247 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 19 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.407+0000 7fcd46400700 1 -- 172.21.15.44:0/3406258943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd40072820 msgr2=0x7fcd40072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.407+0000 7fcd46400700 1 --2- 172.21.15.44:0/3406258943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd40072820 0x7fcd40072c30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcd30004940 tx=0x7fcd30009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.408+0000 7fcd46400700 1 -- 172.21.15.44:0/3406258943 shutdown_connections 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.408+0000 7fcd46400700 1 --2- 172.21.15.44:0/3406258943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd40073170 0x7fcd40073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.408+0000 7fcd46400700 1 --2- 172.21.15.44:0/3406258943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd40072820 0x7fcd40072c30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.408+0000 7fcd46400700 1 -- 172.21.15.44:0/3406258943 >> 172.21.15.44:0/3406258943 conn(0x7fcd400fa950 msgr2=0x7fcd400fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.408+0000 7fcd46400700 1 -- 172.21.15.44:0/3406258943 shutdown_connections 2024-04-16T11:50:15.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.408+0000 7fcd46400700 1 -- 172.21.15.44:0/3406258943 wait complete. 2024-04-16T11:50:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.409+0000 7fcd46400700 1 Processor -- start 2024-04-16T11:50:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.409+0000 7fcd46400700 1 -- start start 2024-04-16T11:50:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.409+0000 7fcd46400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd40072820 0x7fcd401140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.409+0000 7fcd46400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 0x7fcd40115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.409+0000 7fcd46400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40114b30 con 0x7fcd401145e0 2024-04-16T11:50:15.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.409+0000 7fcd46400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40114c70 con 0x7fcd40072820 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 0x7fcd40115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 0x7fcd40115430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50768/0 (socket says 172.21.15.44:50768) 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3ea00700 1 -- 172.21.15.44:0/2229011702 learned_addr learned my addr 172.21.15.44:0/2229011702 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3f400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd40072820 0x7fcd401140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3ea00700 1 -- 172.21.15.44:0/2229011702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd40072820 msgr2=0x7fcd401140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3ea00700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd40072820 0x7fcd401140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.410+0000 7fcd3ea00700 1 -- 172.21.15.44:0/2229011702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd30005040 con 0x7fcd401145e0 2024-04-16T11:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.411+0000 7fcd3ea00700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 0x7fcd40115430 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00b790 tx=0x7fcd2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.411+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd2c00f770 con 0x7fcd401145e0 2024-04-16T11:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.411+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd40115b10 con 0x7fcd401145e0 2024-04-16T11:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.411+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd2c00fda0 con 0x7fcd401145e0 2024-04-16T11:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.411+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd2c00d310 con 0x7fcd401145e0 2024-04-16T11:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.411+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd401160d0 con 0x7fcd401145e0 2024-04-16T11:50:15.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.413+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4005f5f0 con 0x7fcd401145e0 2024-04-16T11:50:15.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.414+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcd2c010420 con 0x7fcd401145e0 2024-04-16T11:50:15.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.415+0000 7fcd37400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcd2406ec10 0x7fcd240710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.415+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fcd2c08e950 con 0x7fcd401145e0 2024-04-16T11:50:15.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.415+0000 7fcd3f400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcd2406ec10 0x7fcd240710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:15.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.416+0000 7fcd3f400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcd2406ec10 0x7fcd240710c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000b3b0 tx=0x7fcd30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:15.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.420+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcd2c05b200 con 0x7fcd401145e0 2024-04-16T11:50:15.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.672+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd40073170 con 0x7fcd2406ec10 2024-04-16T11:50:15.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.677+0000 7fcd37400700 1 -- 172.21.15.44:0/2229011702 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcd40073170 con 0x7fcd2406ec10 2024-04-16T11:50:15.675 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:50:15.675 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:50:15.675 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "13/21 daemons upgraded", 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:50:15.676 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:50:15.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.681+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcd2406ec10 msgr2=0x7fcd240710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcd2406ec10 0x7fcd240710c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000b3b0 tx=0x7fcd30009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 msgr2=0x7fcd40115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 0x7fcd40115430 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00b790 tx=0x7fcd2c009250 comp rx=0 tx=0).stop 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 shutdown_connections 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcd2406ec10 0x7fcd240710c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd40072820 0x7fcd401140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 --2- 172.21.15.44:0/2229011702 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd401145e0 0x7fcd40115430 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.682+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 >> 172.21.15.44:0/2229011702 conn(0x7fcd400fa950 msgr2=0x7fcd400fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:15.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.683+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 shutdown_connections 2024-04-16T11:50:15.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.683+0000 7fcd46400700 1 -- 172.21.15.44:0/2229011702 wait complete. 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 -- 172.21.15.44:0/3007456555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 msgr2=0x7f61580fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 --2- 172.21.15.44:0/3007456555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f61580fe550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6144004940 tx=0x7f6144009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 -- 172.21.15.44:0/3007456555 shutdown_connections 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 --2- 172.21.15.44:0/3007456555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f61580fe550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 --2- 172.21.15.44:0/3007456555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580fcef0 0x7f61580fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 -- 172.21.15.44:0/3007456555 >> 172.21.15.44:0/3007456555 conn(0x7f61580f8530 msgr2=0x7f61580fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 -- 172.21.15.44:0/3007456555 shutdown_connections 2024-04-16T11:50:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.844+0000 7f615de00700 1 -- 172.21.15.44:0/3007456555 wait complete. 2024-04-16T11:50:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.845+0000 7f615de00700 1 Processor -- start 2024-04-16T11:50:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.845+0000 7f615de00700 1 -- start start 2024-04-16T11:50:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f615de00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580fcef0 0x7f6158071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f615de00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f6158070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:15.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f615de00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580708a0 con 0x7f61580fe0e0 2024-04-16T11:50:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f615de00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580709e0 con 0x7f61580fcef0 2024-04-16T11:50:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f6156000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f6158070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f6156000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f6158070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50788/0 (socket says 172.21.15.44:50788) 2024-04-16T11:50:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.846+0000 7f6156000700 1 -- 172.21.15.44:0/1817131616 learned_addr learned my addr 172.21.15.44:0/1817131616 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:15.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.847+0000 7f6156a00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580fcef0 0x7f6158071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.847+0000 7f6156000700 1 -- 172.21.15.44:0/1817131616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580fcef0 msgr2=0x7f6158071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.847+0000 7f6156000700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580fcef0 0x7f6158071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.847+0000 7f6156000700 1 -- 172.21.15.44:0/1817131616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144005040 con 0x7f61580fe0e0 2024-04-16T11:50:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.847+0000 7f6156000700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f6158070310 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6144009150 tx=0x7f6144009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.848+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614400e040 con 0x7f61580fe0e0 2024-04-16T11:50:15.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.848+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f614400bbf0 con 0x7f61580fe0e0 2024-04-16T11:50:15.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.848+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614401c630 con 0x7f61580fe0e0 2024-04-16T11:50:15.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.848+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6158070c60 con 0x7f61580fe0e0 2024-04-16T11:50:15.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.848+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6158115ac0 con 0x7f61580fe0e0 2024-04-16T11:50:15.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.849+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615805f5f0 con 0x7f61580fe0e0 2024-04-16T11:50:15.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.851+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f614400bd60 con 0x7f61580fe0e0 2024-04-16T11:50:15.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.852+0000 7f614ea00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f613c06ec10 0x7f613c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.852+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f614408fd80 con 0x7f61580fe0e0 2024-04-16T11:50:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.852+0000 7f6156a00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f613c06ec10 0x7f613c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:15.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.853+0000 7f6156a00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f613c06ec10 0x7f613c0710c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6148007fd0 tx=0x7f6148009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:15.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:15.857+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6144058a40 con 0x7f61580fe0e0 2024-04-16T11:50:15.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2761511118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/422042891' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:15 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi044.hlafhv"]}]: dispatch 2024-04-16T11:50:16.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2761511118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/422042891' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:15 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi044.hlafhv"]}]: dispatch 2024-04-16T11:50:16.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.179+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f615804bed0 con 0x7f61580fe0e0 2024-04-16T11:50:16.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.180+0000 7f614ea00700 1 -- 172.21.15.44:0/1817131616 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f614405c100 con 0x7f61580fe0e0 2024-04-16T11:50:16.178 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:50:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f613c06ec10 msgr2=0x7f613c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f613c06ec10 0x7f613c0710c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6148007fd0 tx=0x7f6148009040 comp rx=0 tx=0).stop 2024-04-16T11:50:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 msgr2=0x7f6158070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f6158070310 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6144009150 tx=0x7f6144009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 shutdown_connections 2024-04-16T11:50:16.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f613c06ec10 0x7f613c0710c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:16.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61580fcef0 0x7f6158071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:16.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 --2- 172.21.15.44:0/1817131616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f61580fe0e0 0x7f6158070310 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:16.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.183+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 >> 172.21.15.44:0/1817131616 conn(0x7f61580f8530 msgr2=0x7f61580f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:16.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.184+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 shutdown_connections 2024-04-16T11:50:16.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:16.184+0000 7f615de00700 1 -- 172.21.15.44:0/1817131616 wait complete. 2024-04-16T11:50:16.904 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:16 smithi044 ceph-mon[93020]: Detected new or changed devices on smithi044 2024-04-16T11:50:16.905 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:16 smithi044 ceph-mon[93020]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:16.905 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:16 smithi044 ceph-mon[93020]: Upgrade: It appears safe to stop mds.cephfs.smithi044.hlafhv 2024-04-16T11:50:16.905 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:16 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1817131616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:50:17.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:16 smithi045 ceph-mon[81423]: Detected new or changed devices on smithi044 2024-04-16T11:50:17.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:16 smithi045 ceph-mon[81423]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:17.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:16 smithi045 ceph-mon[81423]: Upgrade: It appears safe to stop mds.cephfs.smithi044.hlafhv 2024-04-16T11:50:17.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:16 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1817131616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:50:18.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:17 smithi045 ceph-mon[81423]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:18.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:17 smithi044 ceph-mon[93020]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:19.186 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:18 smithi044 ceph-mon[93020]: Upgrade: Updating mds.cephfs.smithi044.hlafhv 2024-04-16T11:50:19.186 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:18 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:19.186 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:18 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:50:19.186 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:18 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:19.186 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:18 smithi044 ceph-mon[93020]: Deploying daemon mds.cephfs.smithi044.hlafhv on smithi044 2024-04-16T11:50:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:18 smithi045 ceph-mon[81423]: Upgrade: Updating mds.cephfs.smithi044.hlafhv 2024-04-16T11:50:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:18 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:18 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi044.hlafhv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:50:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:18 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:18 smithi045 ceph-mon[81423]: Deploying daemon mds.cephfs.smithi044.hlafhv on smithi044 2024-04-16T11:50:20.153 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:19 smithi044 ceph-mon[93020]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:19 smithi045 ceph-mon[81423]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:21.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 526, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105547 2024-04-16T11:50:21.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 526, success, outsize: 120 2024-04-16T11:50:21.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 414, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122895 2024-04-16T11:50:21.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 414, success, outsize: 120 2024-04-16T11:50:21.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[93016]: 2024-04-16T11:50:21.138+0000 7f2289600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:50:22.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 416, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122896 2024-04-16T11:50:22.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 416, success, outsize: 120 2024-04-16T11:50:22.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 528, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105558 2024-04-16T11:50:22.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 528, success, outsize: 120 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: osdmap e89: 6 total, 6 up, 6 in 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: Standby daemon mds.cephfs.smithi045.noofsd assigned to filesystem cephfs as rank 0 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi045.noofsd=up:replay} 2 up:standby 2024-04-16T11:50:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:50:22.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:22.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: osdmap e89: 6 total, 6 up, 6 in 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: Standby daemon mds.cephfs.smithi045.noofsd assigned to filesystem cephfs as rank 0 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi045.noofsd=up:replay} 2 up:standby 2024-04-16T11:50:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:50:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:23 smithi044 ceph-mon[93020]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00399994 2024-04-16T11:50:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:23 smithi044 ceph-mon[93020]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00499992 2024-04-16T11:50:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:23 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:reconnect 2024-04-16T11:50:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:23 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi045.noofsd=up:reconnect} 2 up:standby 2024-04-16T11:50:23.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:23 smithi044 ceph-mon[93020]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:23 smithi045 ceph-mon[81423]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00399994 2024-04-16T11:50:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:23 smithi045 ceph-mon[81423]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00499992 2024-04-16T11:50:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:23 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:reconnect 2024-04-16T11:50:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:23 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi045.noofsd=up:reconnect} 2 up:standby 2024-04-16T11:50:23.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:23 smithi045 ceph-mon[81423]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:24.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:24 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:rejoin 2024-04-16T11:50:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:24 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi045.noofsd=up:rejoin} 2 up:standby 2024-04-16T11:50:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:24 smithi044 ceph-mon[93020]: daemon mds.cephfs.smithi045.noofsd is now active in filesystem cephfs as rank 0 2024-04-16T11:50:24.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:24 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:rejoin 2024-04-16T11:50:24.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:24 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi045.noofsd=up:rejoin} 2 up:standby 2024-04-16T11:50:24.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:24 smithi045 ceph-mon[81423]: daemon mds.cephfs.smithi045.noofsd is now active in filesystem cephfs as rank 0 2024-04-16T11:50:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:25 smithi044 ceph-mon[93020]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:50:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:25 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:50:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:25 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:active 2024-04-16T11:50:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:25 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi045.noofsd=up:active} 2 up:standby 2024-04-16T11:50:25.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:25 smithi044 ceph-mon[93020]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2024-04-16T11:50:25.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:25 smithi045 ceph-mon[81423]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:50:25.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:25 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:50:25.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:25 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] up:active 2024-04-16T11:50:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:25 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi045.noofsd=up:active} 2 up:standby 2024-04-16T11:50:25.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:25 smithi045 ceph-mon[81423]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2024-04-16T11:50:26.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:26.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:27.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:27 smithi044 ceph-mon[93020]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T11:50:27.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:27 smithi045 ceph-mon[81423]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T11:50:29.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:29 smithi044 ceph-mon[93020]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.8 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T11:50:29.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:29 smithi045 ceph-mon[81423]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.8 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T11:50:31.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:31 smithi044 ceph-mon[93020]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:50:31.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:31 smithi045 ceph-mon[81423]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:50:33.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:33 smithi044 ceph-mon[93020]: pgmap v220: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.1 KiB/s rd, 183 B/s wr, 4 op/s 2024-04-16T11:50:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:33 smithi045 ceph-mon[81423]: pgmap v220: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.1 KiB/s rd, 183 B/s wr, 4 op/s 2024-04-16T11:50:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:33.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:34 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] up:boot 2024-04-16T11:50:34.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:34 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi045.noofsd=up:active} 3 up:standby 2024-04-16T11:50:34.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:50:34.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:34 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] up:boot 2024-04-16T11:50:34.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:34 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi045.noofsd=up:active} 3 up:standby 2024-04-16T11:50:34.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:50:35.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:35 smithi044 ceph-mon[93020]: pgmap v221: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:35 smithi045 ceph-mon[81423]: pgmap v221: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:37.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:37 smithi044 ceph-mon[93020]: pgmap v222: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:37.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:37 smithi045 ceph-mon[81423]: pgmap v222: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:50:39.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:39 smithi044 ceph-mon[93020]: pgmap v223: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-16T11:50:39.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:39 smithi045 ceph-mon[81423]: pgmap v223: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-16T11:50:40.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:40.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:40.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:40.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:40.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:41.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:41 smithi044 ceph-mon[93020]: pgmap v224: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 746 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T11:50:41.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:41 smithi045 ceph-mon[81423]: pgmap v224: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 746 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T11:50:42.889 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:42 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:42.889 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:42 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:43.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:42 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:43.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:42 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:43.930 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:43 smithi044 ceph-mon[93020]: pgmap v225: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T11:50:44.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:43 smithi045 ceph-mon[81423]: pgmap v225: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-16T11:50:45.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 418, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124119 2024-04-16T11:50:45.843 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 418, success, outsize: 120 2024-04-16T11:50:45.865 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 420, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124212 2024-04-16T11:50:45.865 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 420, success, outsize: 120 2024-04-16T11:50:45.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 422, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124308 2024-04-16T11:50:45.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 422, success, outsize: 120 2024-04-16T11:50:45.914 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:45 smithi044 ceph-mon[93020]: pgmap v226: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:45.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 424, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124410 2024-04-16T11:50:45.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 424, success, outsize: 120 2024-04-16T11:50:45.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 426, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124509 2024-04-16T11:50:45.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 426, success, outsize: 120 2024-04-16T11:50:45.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 428, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124602 2024-04-16T11:50:45.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 428, success, outsize: 120 2024-04-16T11:50:45.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 430, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124698 2024-04-16T11:50:45.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 430, success, outsize: 120 2024-04-16T11:50:46.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 432, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124800 2024-04-16T11:50:46.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 432, success, outsize: 120 2024-04-16T11:50:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 434, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124899 2024-04-16T11:50:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 434, success, outsize: 120 2024-04-16T11:50:46.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 436, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 124994 2024-04-16T11:50:46.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 436, success, outsize: 120 2024-04-16T11:50:46.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125091 2024-04-16T11:50:46.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 438, success, outsize: 120 2024-04-16T11:50:46.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:45 smithi045 ceph-mon[81423]: pgmap v226: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:46.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 440, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125193 2024-04-16T11:50:46.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 440, success, outsize: 120 2024-04-16T11:50:46.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 442, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125292 2024-04-16T11:50:46.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 442, success, outsize: 120 2024-04-16T11:50:46.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 444, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125385 2024-04-16T11:50:46.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 444, success, outsize: 120 2024-04-16T11:50:46.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 446, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125481 2024-04-16T11:50:46.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 446, success, outsize: 120 2024-04-16T11:50:46.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 448, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125583 2024-04-16T11:50:46.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 448, success, outsize: 120 2024-04-16T11:50:46.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 450, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125682 2024-04-16T11:50:46.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 450, success, outsize: 120 2024-04-16T11:50:46.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 452, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125775 2024-04-16T11:50:46.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 452, success, outsize: 120 2024-04-16T11:50:46.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 454, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125875 2024-04-16T11:50:46.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 454, success, outsize: 120 2024-04-16T11:50:46.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 456, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125978 2024-04-16T11:50:46.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 456, success, outsize: 120 2024-04-16T11:50:46.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 458, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126080 2024-04-16T11:50:46.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 458, success, outsize: 120 2024-04-16T11:50:46.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 460, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126173 2024-04-16T11:50:46.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 460, success, outsize: 120 2024-04-16T11:50:46.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 462, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126269 2024-04-16T11:50:46.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 462, success, outsize: 120 2024-04-16T11:50:46.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 464, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126371 2024-04-16T11:50:46.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 464, success, outsize: 120 2024-04-16T11:50:46.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 466, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126471 2024-04-16T11:50:46.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 466, success, outsize: 120 2024-04-16T11:50:46.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 468, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126574 2024-04-16T11:50:46.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 468, success, outsize: 120 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 -- 172.21.15.44:0/559530174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 msgr2=0x7f3d940fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 --2- 172.21.15.44:0/559530174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d940fd330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3d88004940 tx=0x7f3d88009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 -- 172.21.15.44:0/559530174 shutdown_connections 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 --2- 172.21.15.44:0/559530174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d940fe090 0x7f3d940fe500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 --2- 172.21.15.44:0/559530174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d940fd330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 -- 172.21.15.44:0/559530174 >> 172.21.15.44:0/559530174 conn(0x7f3d940f85c0 msgr2=0x7f3d940faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 -- 172.21.15.44:0/559530174 shutdown_connections 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.362+0000 7f3d9c400700 1 -- 172.21.15.44:0/559530174 wait complete. 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.363+0000 7f3d9c400700 1 Processor -- start 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.363+0000 7f3d9c400700 1 -- start start 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d9c400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d9410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d9c400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d940fe090 0x7f3d941101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d9c400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d941107a0 con 0x7f3d940fcf20 2024-04-16T11:50:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d9c400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94110ff0 con 0x7f3d940fe090 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d99a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d9410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d99a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d9410fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60622/0 (socket says 172.21.15.44:60622) 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.364+0000 7f3d99a00700 1 -- 172.21.15.44:0/2392029960 learned_addr learned my addr 172.21.15.44:0/2392029960 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.365+0000 7f3d99000700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d940fe090 0x7f3d941101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.365+0000 7f3d99a00700 1 -- 172.21.15.44:0/2392029960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d940fe090 msgr2=0x7f3d941101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.365+0000 7f3d99a00700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d940fe090 0x7f3d941101a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.365+0000 7f3d99a00700 1 -- 172.21.15.44:0/2392029960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88005040 con 0x7f3d940fcf20 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.366+0000 7f3d99a00700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d9410fc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800b970 tx=0x7f3d88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.366+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8800e040 con 0x7f3d940fcf20 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.366+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d88004290 con 0x7f3d940fcf20 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.366+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d94111270 con 0x7f3d940fcf20 2024-04-16T11:50:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.366+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8801c630 con 0x7f3d940fcf20 2024-04-16T11:50:46.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.366+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d941117b0 con 0x7f3d940fcf20 2024-04-16T11:50:46.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.367+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d94101a20 con 0x7f3d940fcf20 2024-04-16T11:50:46.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.370+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3d8800bbf0 con 0x7f3d940fcf20 2024-04-16T11:50:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.371+0000 7f3d91600700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3d7c06ec10 0x7f3d7c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.371+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f3d8808fcc0 con 0x7f3d940fcf20 2024-04-16T11:50:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.372+0000 7f3d99000700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3d7c06ec10 0x7f3d7c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.373+0000 7f3d99000700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3d7c06ec10 0x7f3d7c0710c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3d80004f40 tx=0x7f3d8000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.377+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3d88095020 con 0x7f3d940fcf20 2024-04-16T11:50:46.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 470, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126679 2024-04-16T11:50:46.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 470, success, outsize: 120 2024-04-16T11:50:46.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 472, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126781 2024-04-16T11:50:46.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 472, success, outsize: 120 2024-04-16T11:50:46.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.650+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d94111a90 con 0x7f3d7c06ec10 2024-04-16T11:50:46.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.653+0000 7f3d91600700 1 -- 172.21.15.44:0/2392029960 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3d94111a90 con 0x7f3d7c06ec10 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3d7c06ec10 msgr2=0x7f3d7c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3d7c06ec10 0x7f3d7c0710c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3d80004f40 tx=0x7f3d8000b040 comp rx=0 tx=0).stop 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 msgr2=0x7f3d9410fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d9410fc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800b970 tx=0x7f3d88009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 shutdown_connections 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3d7c06ec10 0x7f3d7c0710c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3d940fcf20 0x7f3d9410fc60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 --2- 172.21.15.44:0/2392029960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d940fe090 0x7f3d941101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 >> 172.21.15.44:0/2392029960 conn(0x7f3d940f85c0 msgr2=0x7f3d940f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:46.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.656+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 shutdown_connections 2024-04-16T11:50:46.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.657+0000 7f3d9c400700 1 -- 172.21.15.44:0/2392029960 wait complete. 2024-04-16T11:50:46.671 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.820+0000 7f8b16400700 1 -- 172.21.15.44:0/4145576487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fe0e0 msgr2=0x7f8b100fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.820+0000 7f8b16400700 1 --2- 172.21.15.44:0/4145576487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fe0e0 0x7f8b100fe550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8af8004940 tx=0x7f8af8009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.820+0000 7f8b16400700 1 -- 172.21.15.44:0/4145576487 shutdown_connections 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.820+0000 7f8b16400700 1 --2- 172.21.15.44:0/4145576487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fe0e0 0x7f8b100fe550 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.820+0000 7f8b16400700 1 --2- 172.21.15.44:0/4145576487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b100fcef0 0x7f8b100fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.820+0000 7f8b16400700 1 -- 172.21.15.44:0/4145576487 >> 172.21.15.44:0/4145576487 conn(0x7f8b100f8530 msgr2=0x7f8b100fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.821+0000 7f8b16400700 1 -- 172.21.15.44:0/4145576487 shutdown_connections 2024-04-16T11:50:46.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.821+0000 7f8b16400700 1 -- 172.21.15.44:0/4145576487 wait complete. 2024-04-16T11:50:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.821+0000 7f8b16400700 1 Processor -- start 2024-04-16T11:50:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b16400700 1 -- start start 2024-04-16T11:50:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b16400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 0x7f8b10071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b16400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b100fe0e0 0x7f8b10070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b16400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100708a0 con 0x7f8b100fcef0 2024-04-16T11:50:46.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b16400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100709e0 con 0x7f8b100fe0e0 2024-04-16T11:50:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b0f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 0x7f8b10071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b0f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 0x7f8b10071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60636/0 (socket says 172.21.15.44:60636) 2024-04-16T11:50:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.822+0000 7f8b0f400700 1 -- 172.21.15.44:0/2290567485 learned_addr learned my addr 172.21.15.44:0/2290567485 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:46.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.823+0000 7f8b07e00700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b100fe0e0 0x7f8b10070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.823+0000 7f8b0f400700 1 -- 172.21.15.44:0/2290567485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b100fe0e0 msgr2=0x7f8b10070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.823+0000 7f8b0f400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b100fe0e0 0x7f8b10070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.823+0000 7f8b0f400700 1 -- 172.21.15.44:0/2290567485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af8005040 con 0x7f8b100fcef0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.823+0000 7f8b0f400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 0x7f8b10071df0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b00002a80 tx=0x7f8b00004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.824+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b00007cc0 con 0x7f8b100fcef0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.824+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b0000f040 con 0x7f8b100fcef0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.824+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b0000da30 con 0x7f8b100fcef0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.824+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b10070cc0 con 0x7f8b100fcef0 2024-04-16T11:50:46.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.824+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b10115ac0 con 0x7f8b100fcef0 2024-04-16T11:50:46.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.825+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1005f5f0 con 0x7f8b100fcef0 2024-04-16T11:50:46.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.826+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8b0000db90 con 0x7f8b100fcef0 2024-04-16T11:50:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.827+0000 7f8b07400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8af406ec10 0x7f8af40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.827+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f8b0008e850 con 0x7f8b100fcef0 2024-04-16T11:50:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.827+0000 7f8b07e00700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8af406ec10 0x7f8af40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:46.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.828+0000 7f8b07e00700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8af406ec10 0x7f8af40710c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8b100723d0 tx=0x7f8af8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:46.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:46.832+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8b000573b0 con 0x7f8b100fcef0 2024-04-16T11:50:47.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.105+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b101024a0 con 0x7f8af406ec10 2024-04-16T11:50:47.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.108+0000 7f8b07400700 1 -- 172.21.15.44:0/2290567485 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8b101024a0 con 0x7f8af406ec10 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8af406ec10 msgr2=0x7f8af40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8af406ec10 0x7f8af40710c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8b100723d0 tx=0x7f8af8009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 msgr2=0x7f8b10071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 0x7f8b10071df0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b00002a80 tx=0x7f8b00004600 comp rx=0 tx=0).stop 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 shutdown_connections 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8af406ec10 0x7f8af40710c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b100fcef0 0x7f8b10071df0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 --2- 172.21.15.44:0/2290567485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b100fe0e0 0x7f8b10070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 >> 172.21.15.44:0/2290567485 conn(0x7f8b100f8530 msgr2=0x7f8b100f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:47.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 shutdown_connections 2024-04-16T11:50:47.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.112+0000 7f8b16400700 1 -- 172.21.15.44:0/2290567485 wait complete. 2024-04-16T11:50:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.267+0000 7f24f6600700 1 -- 172.21.15.44:0/729186958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 msgr2=0x7f24f00703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.267+0000 7f24f6600700 1 --2- 172.21.15.44:0/729186958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f00703e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f24e0004970 tx=0x7f24e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:47.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.268+0000 7f24f6600700 1 -- 172.21.15.44:0/729186958 shutdown_connections 2024-04-16T11:50:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.268+0000 7f24f6600700 1 --2- 172.21.15.44:0/729186958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 0x7f24f0070d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.268+0000 7f24f6600700 1 --2- 172.21.15.44:0/729186958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f00703e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.268+0000 7f24f6600700 1 -- 172.21.15.44:0/729186958 >> 172.21.15.44:0/729186958 conn(0x7f24f00f8530 msgr2=0x7f24f00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.268+0000 7f24f6600700 1 -- 172.21.15.44:0/729186958 shutdown_connections 2024-04-16T11:50:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.268+0000 7f24f6600700 1 -- 172.21.15.44:0/729186958 wait complete. 2024-04-16T11:50:47.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24f6600700 1 Processor -- start 2024-04-16T11:50:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24f6600700 1 -- start start 2024-04-16T11:50:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24f6600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 0x7f24f010fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24f6600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f0110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24f6600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0110750 con 0x7f24f0071ed0 2024-04-16T11:50:47.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24f6600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0110fa0 con 0x7f24f0070920 2024-04-16T11:50:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24eea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f0110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24eea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f0110150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60660/0 (socket says 172.21.15.44:60660) 2024-04-16T11:50:47.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.269+0000 7f24eea00700 1 -- 172.21.15.44:0/3354325557 learned_addr learned my addr 172.21.15.44:0/3354325557 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.270+0000 7f24ef400700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 0x7f24f010fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.270+0000 7f24eea00700 1 -- 172.21.15.44:0/3354325557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 msgr2=0x7f24f010fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.270+0000 7f24eea00700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 0x7f24f010fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.270+0000 7f24eea00700 1 -- 172.21.15.44:0/3354325557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0005040 con 0x7f24f0071ed0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.270+0000 7f24ef400700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 0x7f24f010fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.270+0000 7f24eea00700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f0110150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c810 tx=0x7f24dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.271+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24dc012810 con 0x7f24f0071ed0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.271+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24dc012e40 con 0x7f24f0071ed0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.271+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f0111280 con 0x7f24f0071ed0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.271+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24dc00a370 con 0x7f24f0071ed0 2024-04-16T11:50:47.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.271+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f01117c0 con 0x7f24f0071ed0 2024-04-16T11:50:47.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.272+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f0109560 con 0x7f24f0071ed0 2024-04-16T11:50:47.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.273+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24dc011810 con 0x7f24f0071ed0 2024-04-16T11:50:47.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.275+0000 7f24e7400700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f24d406ec10 0x7f24d40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:47.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.275+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f24dc08ef30 con 0x7f24f0071ed0 2024-04-16T11:50:47.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.275+0000 7f24ef400700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f24d406ec10 0x7f24d40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:47.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.276+0000 7f24ef400700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f24d406ec10 0x7f24d40710c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24e0004940 tx=0x7f24e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:47.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.280+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f24dc05b5c0 con 0x7f24f0071ed0 2024-04-16T11:50:47.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.521+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f24f0111b80 con 0x7f24d406ec10 2024-04-16T11:50:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.552+0000 7f24e7400700 1 -- 172.21.15.44:0/3354325557 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f24f0111b80 con 0x7f24d406ec10 2024-04-16T11:50:47.549 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:50:47.549 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (7m) 8s ago 18m 13.6M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:50:47.549 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (4m) 8s ago 18m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (4m) 75s ago 17m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (7m) 8s ago 17m 73.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (15s) 8s ago 11m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (48s) 8s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (11m) 75s ago 11m 14.1M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (11m) 75s ago 11m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 6cf7aba15fc8 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (7m) 8s ago 20m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (6m) 75s ago 17m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (6m) 8s ago 20m 52.0M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (5m) 75s ago 16m 36.1M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (8m) 8s ago 18m 7708k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (7m) 75s ago 17m 7746k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:50:47.550 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (4m) 8s ago 16m 54.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:50:47.551 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (3m) 8s ago 15m 44.8M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:50:47.551 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (3m) 8s ago 14m 44.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:50:47.551 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (2m) 75s ago 14m 42.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:50:47.551 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (2m) 75s ago 13m 41.8M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:50:47.551 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (79s) 75s ago 13m 11.1M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:50:47.551 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (6m) 8s ago 17m 57.9M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:50:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f24d406ec10 msgr2=0x7f24d40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f24d406ec10 0x7f24d40710c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24e0004940 tx=0x7f24e0009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 msgr2=0x7f24f0110150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f0110150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c810 tx=0x7f24dc004470 comp rx=0 tx=0).stop 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 shutdown_connections 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f24d406ec10 0x7f24d40710c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24f0070920 0x7f24f010fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 --2- 172.21.15.44:0/3354325557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f24f0071ed0 0x7f24f0110150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.555+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 >> 172.21.15.44:0/3354325557 conn(0x7f24f00f8530 msgr2=0x7f24f0102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.556+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 shutdown_connections 2024-04-16T11:50:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.556+0000 7f24f6600700 1 -- 172.21.15.44:0/3354325557 wait complete. 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 -- 172.21.15.44:0/2020418077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 msgr2=0x7fbaa40ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 --2- 172.21.15.44:0/2020418077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa40ff310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba94004940 tx=0x7fba94009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 -- 172.21.15.44:0/2020418077 shutdown_connections 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 --2- 172.21.15.44:0/2020418077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 0x7fbaa41004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 --2- 172.21.15.44:0/2020418077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa40ff310 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 -- 172.21.15.44:0/2020418077 >> 172.21.15.44:0/2020418077 conn(0x7fbaa40fa790 msgr2=0x7fbaa40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:47.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 -- 172.21.15.44:0/2020418077 shutdown_connections 2024-04-16T11:50:47.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.719+0000 7fbaa3200700 1 -- 172.21.15.44:0/2020418077 wait complete. 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.720+0000 7fbaa3200700 1 Processor -- start 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.720+0000 7fbaa3200700 1 -- start start 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.720+0000 7fbaa3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa410fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.720+0000 7fbaa3200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 0x7fbaa410ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.720+0000 7fbaa3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4110650 con 0x7fbaa40fef00 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.720+0000 7fbaa3200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4110e40 con 0x7fbaa4100070 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa410fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa410fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60668/0 (socket says 172.21.15.44:60668) 2024-04-16T11:50:47.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 -- 172.21.15.44:0/3712131460 learned_addr learned my addr 172.21.15.44:0/3712131460 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1400700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 0x7fbaa410ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 -- 172.21.15.44:0/3712131460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 msgr2=0x7fbaa410ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 0x7fbaa410ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 -- 172.21.15.44:0/3712131460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba94005040 con 0x7fbaa40fef00 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1400700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 0x7fbaa410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.721+0000 7fbaa1e00700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa410fa80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fba9400b3b0 tx=0x7fba94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.722+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba9400e040 con 0x7fbaa40fef00 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.722+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba9400bcc0 con 0x7fbaa40fef00 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.722+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa41110c0 con 0x7fbaa40fef00 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.722+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba9401c630 con 0x7fbaa40fef00 2024-04-16T11:50:47.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.722+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa4111680 con 0x7fbaa40fef00 2024-04-16T11:50:47.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.723+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa40fe290 con 0x7fbaa40fef00 2024-04-16T11:50:47.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.724+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fba94013980 con 0x7fbaa40fef00 2024-04-16T11:50:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.725+0000 7fba9a000700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fba8806ec00 0x7fba880710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.725+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fba9408f750 con 0x7fbaa40fef00 2024-04-16T11:50:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.725+0000 7fbaa1400700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fba8806ec00 0x7fba880710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:47.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.726+0000 7fbaa1400700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fba8806ec00 0x7fba880710b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fba90006fd0 tx=0x7fba90008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:47.730+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fba9405be80 con 0x7fbaa40fef00 2024-04-16T11:50:47.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: pgmap v227: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:47.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:47.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:47.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi045.noofsd"]}]: dispatch 2024-04-16T11:50:47.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: pgmap v227: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:47.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:47.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:47.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:47.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi045.noofsd"]}]: dispatch 2024-04-16T11:50:48.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.062+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbaa41079e0 con 0x7fbaa40fef00 2024-04-16T11:50:48.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.063+0000 7fba9a000700 1 -- 172.21.15.44:0/3712131460 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fba9405ba10 con 0x7fbaa40fef00 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:50:48.061 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 12 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:50:48.062 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:50:48.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fba8806ec00 msgr2=0x7fba880710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fba8806ec00 0x7fba880710b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fba90006fd0 tx=0x7fba90008040 comp rx=0 tx=0).stop 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 msgr2=0x7fbaa410fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa410fa80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fba9400b3b0 tx=0x7fba94009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 shutdown_connections 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fba8806ec00 0x7fba880710b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaa40fef00 0x7fbaa410fa80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 --2- 172.21.15.44:0/3712131460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa4100070 0x7fbaa410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.066+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 >> 172.21.15.44:0/3712131460 conn(0x7fbaa40fa790 msgr2=0x7fbaa40fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.067+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 shutdown_connections 2024-04-16T11:50:48.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.067+0000 7fbaa3200700 1 -- 172.21.15.44:0/3712131460 wait complete. 2024-04-16T11:50:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.222+0000 7f59a0c00700 1 -- 172.21.15.44:0/797070533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 msgr2=0x7f599c0ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.222+0000 7f59a0c00700 1 --2- 172.21.15.44:0/797070533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c0ff430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f598c004940 tx=0x7f598c009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.223+0000 7f59a0c00700 1 -- 172.21.15.44:0/797070533 shutdown_connections 2024-04-16T11:50:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.223+0000 7f59a0c00700 1 --2- 172.21.15.44:0/797070533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f599c1002d0 0x7f599c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.223+0000 7f59a0c00700 1 --2- 172.21.15.44:0/797070533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c0ff430 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.223+0000 7f59a0c00700 1 -- 172.21.15.44:0/797070533 >> 172.21.15.44:0/797070533 conn(0x7f599c0fa950 msgr2=0x7f599c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:48.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.223+0000 7f59a0c00700 1 -- 172.21.15.44:0/797070533 shutdown_connections 2024-04-16T11:50:48.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.223+0000 7f59a0c00700 1 -- 172.21.15.44:0/797070533 wait complete. 2024-04-16T11:50:48.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f59a0c00700 1 Processor -- start 2024-04-16T11:50:48.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f59a0c00700 1 -- start start 2024-04-16T11:50:48.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f59a0c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c10fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:48.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f59a0c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f599c1002d0 0x7f599c110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:48.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f59a0c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c110720 con 0x7f599c0ff020 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f59a0c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c1150c0 con 0x7f599c1002d0 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f5999e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c10fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f5999e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c10fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60680/0 (socket says 172.21.15.44:60680) 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.224+0000 7f5999e00700 1 -- 172.21.15.44:0/1358961797 learned_addr learned my addr 172.21.15.44:0/1358961797 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.225+0000 7f5999400700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f599c1002d0 0x7f599c110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.225+0000 7f5999e00700 1 -- 172.21.15.44:0/1358961797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f599c1002d0 msgr2=0x7f599c110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.225+0000 7f5999e00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f599c1002d0 0x7f599c110090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.225+0000 7f5999e00700 1 -- 172.21.15.44:0/1358961797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c005040 con 0x7f599c0ff020 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.225+0000 7f5999e00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c10fb50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f598c00b3b0 tx=0x7f598c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.226+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f598c00e040 con 0x7f599c0ff020 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.226+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f598c004290 con 0x7f599c0ff020 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.226+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f598c01c630 con 0x7f599c0ff020 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.226+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c1152c0 con 0x7f599c0ff020 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.226+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c115830 con 0x7f599c0ff020 2024-04-16T11:50:48.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.227+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c04c710 con 0x7f599c0ff020 2024-04-16T11:50:48.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.228+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f598c00bbf0 con 0x7f599c0ff020 2024-04-16T11:50:48.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.229+0000 7f5992000700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f598006ec00 0x7f59800710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:48.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.229+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f598c08fe40 con 0x7f599c0ff020 2024-04-16T11:50:48.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.229+0000 7f5999400700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f598006ec00 0x7f59800710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:48.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.230+0000 7f5999400700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f598006ec00 0x7f59800710b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5988006fd0 tx=0x7f5988008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:48.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.234+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f598c05c5d0 con 0x7f599c0ff020 2024-04-16T11:50:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.516+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f599c115aa0 con 0x7f599c0ff020 2024-04-16T11:50:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.517+0000 7f5992000700 1 -- 172.21.15.44:0/1358961797 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1891 (secure 0 0 0) 0x7f598c05c160 con 0x7f599c0ff020 2024-04-16T11:50:48.514 INFO:teuthology.orchestra.run.smithi044.stdout:e25 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 24 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:50:48.515 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:50:24.169880+0000 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:50:48.516 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 89 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=24277} 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:50:48.517 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 24277 members: 24277 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{0:24277} state up:active seq 165 join_fscid=1 addr [v2:172.21.15.45:6824/4128280511,v1:172.21.15.45:6825/4128280511] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:50:48.518 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:50:48.519 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:24287} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T11:50:48.519 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:50:48.519 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f598006ec00 msgr2=0x7f59800710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f598006ec00 0x7f59800710b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5988006fd0 tx=0x7f5988008040 comp rx=0 tx=0).stop 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 msgr2=0x7f599c10fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c10fb50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f598c00b3b0 tx=0x7f598c009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 shutdown_connections 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f598006ec00 0x7f59800710b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f599c0ff020 0x7f599c10fb50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 --2- 172.21.15.44:0/1358961797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f599c1002d0 0x7f599c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 >> 172.21.15.44:0/1358961797 conn(0x7f599c0fa950 msgr2=0x7f599c0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:48.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 shutdown_connections 2024-04-16T11:50:48.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.520+0000 7f59a0c00700 1 -- 172.21.15.44:0/1358961797 wait complete. 2024-04-16T11:50:48.521 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-16T11:50:48.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.673+0000 7f59f2200700 1 -- 172.21.15.44:0/1153496801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 msgr2=0x7f59ec104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.673+0000 7f59f2200700 1 --2- 172.21.15.44:0/1153496801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec104b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f59d8004940 tx=0x7f59d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:48.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.673+0000 7f59f2200700 1 -- 172.21.15.44:0/1153496801 shutdown_connections 2024-04-16T11:50:48.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.673+0000 7f59f2200700 1 --2- 172.21.15.44:0/1153496801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec104b40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.673+0000 7f59f2200700 1 --2- 172.21.15.44:0/1153496801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec103420 0x7f59ec103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.673+0000 7f59f2200700 1 -- 172.21.15.44:0/1153496801 >> 172.21.15.44:0/1153496801 conn(0x7f59ec072640 msgr2=0x7f59ec074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.674+0000 7f59f2200700 1 -- 172.21.15.44:0/1153496801 shutdown_connections 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.674+0000 7f59f2200700 1 -- 172.21.15.44:0/1153496801 wait complete. 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59f2200700 1 Processor -- start 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59f2200700 1 -- start start 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59f2200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec103420 0x7f59ec113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59f2200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59f2200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec114b20 con 0x7f59ec1046d0 2024-04-16T11:50:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59f2200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec1194c0 con 0x7f59ec103420 2024-04-16T11:50:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59eaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59eaa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60694/0 (socket says 172.21.15.44:60694) 2024-04-16T11:50:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.675+0000 7f59eaa00700 1 -- 172.21.15.44:0/3562431817 learned_addr learned my addr 172.21.15.44:0/3562431817 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.676+0000 7f59eb400700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec103420 0x7f59ec113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.676+0000 7f59eaa00700 1 -- 172.21.15.44:0/3562431817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec103420 msgr2=0x7f59ec113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.676+0000 7f59eaa00700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec103420 0x7f59ec113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.676+0000 7f59eaa00700 1 -- 172.21.15.44:0/3562431817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8005040 con 0x7f59ec1046d0 2024-04-16T11:50:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.676+0000 7f59eaa00700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec114490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f59d800b3b0 tx=0x7f59d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.677+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d800e040 con 0x7f59ec1046d0 2024-04-16T11:50:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.677+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d8004290 con 0x7f59ec1046d0 2024-04-16T11:50:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.677+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d801c630 con 0x7f59ec1046d0 2024-04-16T11:50:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.677+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec1196c0 con 0x7f59ec1046d0 2024-04-16T11:50:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.677+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec119c30 con 0x7f59ec1046d0 2024-04-16T11:50:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.678+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec04c710 con 0x7f59ec1046d0 2024-04-16T11:50:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.679+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f59d800bbf0 con 0x7f59ec1046d0 2024-04-16T11:50:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.680+0000 7f59e3400700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59d006ec00 0x7f59d00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.680+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f59d808fe40 con 0x7f59ec1046d0 2024-04-16T11:50:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.681+0000 7f59eb400700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59d006ec00 0x7f59d00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.681+0000 7f59eb400700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59d006ec00 0x7f59d00710b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f59dc006fd0 tx=0x7f59dc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:48.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.685+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f59d805c5d0 con 0x7f59ec1046d0 2024-04-16T11:50:48.913 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:48 smithi045 ceph-mon[81423]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:48.913 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:48 smithi045 ceph-mon[81423]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:48.914 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:48 smithi045 ceph-mon[81423]: Upgrade: It appears safe to stop mds.cephfs.smithi045.noofsd 2024-04-16T11:50:48.914 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:48 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3712131460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:48.914 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:48 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1358961797' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:50:48.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.926+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f59ec108970 con 0x7f59d006ec00 2024-04-16T11:50:48.923 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:48 smithi044 ceph-mon[93020]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:48.924 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:48 smithi044 ceph-mon[93020]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:48.924 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:48 smithi044 ceph-mon[93020]: Upgrade: It appears safe to stop mds.cephfs.smithi045.noofsd 2024-04-16T11:50:48.924 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:48 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3712131460' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:50:48.924 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:48 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1358961797' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:50:48.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.928+0000 7f59e3400700 1 -- 172.21.15.44:0/3562431817 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f59ec108970 con 0x7f59d006ec00 2024-04-16T11:50:48.925 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:50:48.925 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "14/21 daemons upgraded", 2024-04-16T11:50:48.926 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T11:50:48.927 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:50:48.927 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:50:48.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59d006ec00 msgr2=0x7f59d00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59d006ec00 0x7f59d00710b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f59dc006fd0 tx=0x7f59dc008040 comp rx=0 tx=0).stop 2024-04-16T11:50:48.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 msgr2=0x7f59ec114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:48.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec114490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f59d800b3b0 tx=0x7f59d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:48.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 shutdown_connections 2024-04-16T11:50:48.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59d006ec00 0x7f59d00710b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59ec103420 0x7f59ec113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 --2- 172.21.15.44:0/3562431817 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59ec1046d0 0x7f59ec114490 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:48.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 >> 172.21.15.44:0/3562431817 conn(0x7f59ec072640 msgr2=0x7f59ec073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:48.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 shutdown_connections 2024-04-16T11:50:48.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:48.931+0000 7f59f2200700 1 -- 172.21.15.44:0/3562431817 wait complete. 2024-04-16T11:50:49.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.080+0000 7f06f4800700 1 -- 172.21.15.44:0/1542128999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 msgr2=0x7f06ec070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:49.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.080+0000 7f06f4800700 1 --2- 172.21.15.44:0/1542128999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec070d60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06dc004940 tx=0x7f06dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:49.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.080+0000 7f06f4800700 1 -- 172.21.15.44:0/1542128999 shutdown_connections 2024-04-16T11:50:49.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.080+0000 7f06f4800700 1 --2- 172.21.15.44:0/1542128999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec070d60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:49.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.080+0000 7f06f4800700 1 --2- 172.21.15.44:0/1542128999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06ec071ea0 0x7f06ec0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:49.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.080+0000 7f06f4800700 1 -- 172.21.15.44:0/1542128999 >> 172.21.15.44:0/1542128999 conn(0x7f06ec0f8530 msgr2=0x7f06ec0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:49.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.081+0000 7f06f4800700 1 -- 172.21.15.44:0/1542128999 shutdown_connections 2024-04-16T11:50:49.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.081+0000 7f06f4800700 1 -- 172.21.15.44:0/1542128999 wait complete. 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.082+0000 7f06f4800700 1 Processor -- start 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.082+0000 7f06f4800700 1 -- start start 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.082+0000 7f06f4800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.082+0000 7f06f4800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06ec071ea0 0x7f06ec114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.082+0000 7f06f4800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec114b50 con 0x7f06ec0708f0 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.082+0000 7f06f4800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1153a0 con 0x7f06ec071ea0 2024-04-16T11:50:49.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60714/0 (socket says 172.21.15.44:60714) 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 -- 172.21.15.44:0/3586263899 learned_addr learned my addr 172.21.15.44:0/3586263899 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1400700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06ec071ea0 0x7f06ec114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 -- 172.21.15.44:0/3586263899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06ec071ea0 msgr2=0x7f06ec114550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06ec071ea0 0x7f06ec114550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 -- 172.21.15.44:0/3586263899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc005040 con 0x7f06ec0708f0 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.083+0000 7f06f1e00700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec114010 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f06e0002a80 tx=0x7f06e0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.084+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e0007cc0 con 0x7f06ec0708f0 2024-04-16T11:50:49.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.084+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06e000f040 con 0x7f06ec0708f0 2024-04-16T11:50:49.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.084+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec115680 con 0x7f06ec0708f0 2024-04-16T11:50:49.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.084+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e000dad0 con 0x7f06ec0708f0 2024-04-16T11:50:49.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.084+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec115bc0 con 0x7f06ec0708f0 2024-04-16T11:50:49.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.085+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec05f5f0 con 0x7f06ec0708f0 2024-04-16T11:50:49.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.086+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f06e000dc30 con 0x7f06ec0708f0 2024-04-16T11:50:49.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.087+0000 7f06e6000700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f06d406ec10 0x7f06d40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:50:49.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.087+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f06e008e7d0 con 0x7f06ec0708f0 2024-04-16T11:50:49.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.087+0000 7f06f1400700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f06d406ec10 0x7f06d40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:50:49.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.088+0000 7f06f1400700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f06d406ec10 0x7f06d40710c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b970 tx=0x7f06dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:50:49.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.092+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f06e005aee0 con 0x7f06ec0708f0 2024-04-16T11:50:49.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.408+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f06ec04bed0 con 0x7f06ec0708f0 2024-04-16T11:50:49.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.409+0000 7f06e6000700 1 -- 172.21.15.44:0/3586263899 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f06e005aa70 con 0x7f06ec0708f0 2024-04-16T11:50:49.406 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:50:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f06d406ec10 msgr2=0x7f06d40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f06d406ec10 0x7f06d40710c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b970 tx=0x7f06dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:50:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 msgr2=0x7f06ec114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:50:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec114010 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f06e0002a80 tx=0x7f06e0004600 comp rx=0 tx=0).stop 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 shutdown_connections 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f06d406ec10 0x7f06d40710c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06ec0708f0 0x7f06ec114010 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 --2- 172.21.15.44:0/3586263899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06ec071ea0 0x7f06ec114550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 >> 172.21.15.44:0/3586263899 conn(0x7f06ec0f8530 msgr2=0x7f06ec0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 shutdown_connections 2024-04-16T11:50:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:50:49.412+0000 7f06f4800700 1 -- 172.21.15.44:0/3586263899 wait complete. 2024-04-16T11:50:49.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:49 smithi044 ceph-mon[93020]: pgmap v228: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:49.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:49 smithi044 ceph-mon[93020]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:49.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:49 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3586263899' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:50:50.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:49 smithi045 ceph-mon[81423]: pgmap v228: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:50.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:49 smithi045 ceph-mon[81423]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:50:50.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:49 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3586263899' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:50:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:50 smithi044 ceph-mon[93020]: Upgrade: Updating mds.cephfs.smithi045.noofsd 2024-04-16T11:50:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:50:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:50 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:51.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:50 smithi044 ceph-mon[93020]: Deploying daemon mds.cephfs.smithi045.noofsd on smithi045 2024-04-16T11:50:51.252 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:50 smithi045 ceph-mon[81423]: Upgrade: Updating mds.cephfs.smithi045.noofsd 2024-04-16T11:50:51.253 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:51.253 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.noofsd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:50:51.253 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:50 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:50:51.253 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:50 smithi045 ceph-mon[81423]: Deploying daemon mds.cephfs.smithi045.noofsd on smithi045 2024-04-16T11:50:52.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:51 smithi044 ceph-mon[93020]: pgmap v229: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:51 smithi045 ceph-mon[81423]: pgmap v229: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:53.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 474, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127002 2024-04-16T11:50:53.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 474, success, outsize: 120 2024-04-16T11:50:53.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 530, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105925 2024-04-16T11:50:53.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 530, success, outsize: 120 2024-04-16T11:50:53.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[93016]: 2024-04-16T11:50:53.138+0000 7f2289600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:50:54.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 476, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127003 2024-04-16T11:50:54.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 476, success, outsize: 120 2024-04-16T11:50:54.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 532, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105936 2024-04-16T11:50:54.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 532, success, outsize: 120 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: pgmap v230: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: osdmap e90: 6 total, 6 up, 6 in 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: Standby daemon mds.cephfs.smithi045.hxlkks assigned to filesystem cephfs as rank 0 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:50:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:53 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi045.hxlkks=up:replay} 2 up:standby 2024-04-16T11:50:54.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: pgmap v230: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:50:54.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:50:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:50:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: osdmap e90: 6 total, 6 up, 6 in 2024-04-16T11:50:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: Standby daemon mds.cephfs.smithi045.hxlkks assigned to filesystem cephfs as rank 0 2024-04-16T11:50:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:50:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:50:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:53 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi045.hxlkks=up:replay} 2 up:standby 2024-04-16T11:50:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:55 smithi044 ceph-mon[93020]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00299995 2024-04-16T11:50:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:55 smithi044 ceph-mon[93020]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00399994 2024-04-16T11:50:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:55 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:reconnect 2024-04-16T11:50:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:55 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi045.hxlkks=up:reconnect} 2 up:standby 2024-04-16T11:50:55.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:55 smithi044 ceph-mon[93020]: pgmap v232: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2024-04-16T11:50:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:55 smithi045 ceph-mon[81423]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00299995 2024-04-16T11:50:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:55 smithi045 ceph-mon[81423]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00399994 2024-04-16T11:50:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:55 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:reconnect 2024-04-16T11:50:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:55 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi045.hxlkks=up:reconnect} 2 up:standby 2024-04-16T11:50:55.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:55 smithi045 ceph-mon[81423]: pgmap v232: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2024-04-16T11:50:56.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:56 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:rejoin 2024-04-16T11:50:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:56 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi045.hxlkks=up:rejoin} 2 up:standby 2024-04-16T11:50:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:56 smithi044 ceph-mon[93020]: daemon mds.cephfs.smithi045.hxlkks is now active in filesystem cephfs as rank 0 2024-04-16T11:50:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:56.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:56 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:rejoin 2024-04-16T11:50:56.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:56 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi045.hxlkks=up:rejoin} 2 up:standby 2024-04-16T11:50:56.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:56 smithi045 ceph-mon[81423]: daemon mds.cephfs.smithi045.hxlkks is now active in filesystem cephfs as rank 0 2024-04-16T11:50:56.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:56 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:50:56.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:56 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:50:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:57 smithi044 ceph-mon[93020]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:50:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:57 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:50:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:57 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:active 2024-04-16T11:50:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:57 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi045.hxlkks=up:active} 2 up:standby 2024-04-16T11:50:57.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:57 smithi044 ceph-mon[93020]: pgmap v233: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T11:50:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:57 smithi045 ceph-mon[81423]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:50:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:57 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:50:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:57 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6826/1837389700,v1:172.21.15.45:6827/1837389700] up:active 2024-04-16T11:50:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:57 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi045.hxlkks=up:active} 2 up:standby 2024-04-16T11:50:57.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:57 smithi045 ceph-mon[81423]: pgmap v233: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T11:50:59.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:50:59 smithi044 ceph-mon[93020]: pgmap v234: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T11:51:00.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:50:59 smithi045 ceph-mon[81423]: pgmap v234: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T11:51:01.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:01 smithi044 ceph-mon[93020]: pgmap v235: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:51:02.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:01 smithi045 ceph-mon[81423]: pgmap v235: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:51:02.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:02 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] up:boot 2024-04-16T11:51:02.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:02 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi045.hxlkks=up:active} 3 up:standby 2024-04-16T11:51:02.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:51:02.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:02.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:02.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:02 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:02 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] up:boot 2024-04-16T11:51:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:02 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi045.hxlkks=up:active} 3 up:standby 2024-04-16T11:51:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:51:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:03.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:02 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:04.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:03 smithi045 ceph-mon[81423]: pgmap v236: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:51:04.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:03 smithi044 ceph-mon[93020]: pgmap v236: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T11:51:06.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:05 smithi045 ceph-mon[81423]: pgmap v237: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 178 B/s wr, 4 op/s 2024-04-16T11:51:06.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:05 smithi044 ceph-mon[93020]: pgmap v237: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 178 B/s wr, 4 op/s 2024-04-16T11:51:07.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:07 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:07.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:07 smithi044 ceph-mon[93020]: pgmap v238: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:51:07.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:07.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:07 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:07.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:07 smithi045 ceph-mon[81423]: pgmap v238: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-16T11:51:09.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:09.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:09.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:09 smithi045 ceph-mon[81423]: pgmap v239: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-16T11:51:09.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:09.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:09.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:09 smithi044 ceph-mon[93020]: pgmap v239: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-16T11:51:11.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:11.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:11.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:11 smithi045 ceph-mon[81423]: pgmap v240: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-16T11:51:11.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:11.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:11.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:11 smithi044 ceph-mon[93020]: pgmap v240: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-16T11:51:12.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 534, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106959 2024-04-16T11:51:12.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 534, success, outsize: 120 2024-04-16T11:51:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 536, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107037 2024-04-16T11:51:12.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 536, success, outsize: 120 2024-04-16T11:51:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 538, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107118 2024-04-16T11:51:12.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 538, success, outsize: 120 2024-04-16T11:51:12.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 540, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107205 2024-04-16T11:51:12.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 540, success, outsize: 120 2024-04-16T11:51:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 542, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107289 2024-04-16T11:51:12.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 542, success, outsize: 120 2024-04-16T11:51:12.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 544, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107367 2024-04-16T11:51:12.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 544, success, outsize: 120 2024-04-16T11:51:12.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 546, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107448 2024-04-16T11:51:12.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 546, success, outsize: 120 2024-04-16T11:51:12.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 548, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107535 2024-04-16T11:51:12.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 548, success, outsize: 120 2024-04-16T11:51:12.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 550, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107619 2024-04-16T11:51:12.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 550, success, outsize: 120 2024-04-16T11:51:12.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 552, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107697 2024-04-16T11:51:12.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 552, success, outsize: 120 2024-04-16T11:51:12.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 554, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107778 2024-04-16T11:51:12.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 554, success, outsize: 120 2024-04-16T11:51:12.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 556, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107865 2024-04-16T11:51:12.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 556, success, outsize: 120 2024-04-16T11:51:12.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 558, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107949 2024-04-16T11:51:12.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 558, success, outsize: 120 2024-04-16T11:51:12.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 560, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108027 2024-04-16T11:51:12.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 560, success, outsize: 120 2024-04-16T11:51:12.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 562, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108108 2024-04-16T11:51:12.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 562, success, outsize: 120 2024-04-16T11:51:12.670 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 564, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108195 2024-04-16T11:51:12.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 564, success, outsize: 120 2024-04-16T11:51:12.692 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 566, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108279 2024-04-16T11:51:12.692 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 566, success, outsize: 120 2024-04-16T11:51:12.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 568, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108357 2024-04-16T11:51:12.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 568, success, outsize: 120 2024-04-16T11:51:12.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 570, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108438 2024-04-16T11:51:12.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 570, success, outsize: 120 2024-04-16T11:51:12.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 572, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108525 2024-04-16T11:51:12.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 572, success, outsize: 120 2024-04-16T11:51:12.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 574, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108609 2024-04-16T11:51:12.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 574, success, outsize: 120 2024-04-16T11:51:12.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 576, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108687 2024-04-16T11:51:12.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 576, success, outsize: 120 2024-04-16T11:51:12.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 578, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108768 2024-04-16T11:51:12.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 578, success, outsize: 120 2024-04-16T11:51:12.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 580, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108855 2024-04-16T11:51:12.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 580, success, outsize: 120 2024-04-16T11:51:12.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 582, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108939 2024-04-16T11:51:12.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 582, success, outsize: 120 2024-04-16T11:51:12.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 584, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109017 2024-04-16T11:51:12.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 584, success, outsize: 120 2024-04-16T11:51:12.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 586, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109098 2024-04-16T11:51:12.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 586, success, outsize: 120 2024-04-16T11:51:12.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 588, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109185 2024-04-16T11:51:12.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 588, success, outsize: 120 2024-04-16T11:51:13.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:13 smithi044 ceph-mon[93020]: pgmap v241: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:14.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:13 smithi045 ceph-mon[81423]: pgmap v241: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:15.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: Detected new or changed devices on smithi045 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:14 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi045.hxlkks"]}]: dispatch 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: Detected new or changed devices on smithi045 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.177 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.178 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:15.178 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:14 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi045.hxlkks"]}]: dispatch 2024-04-16T11:51:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:15 smithi044 ceph-mon[93020]: Upgrade: It appears safe to stop mds.cephfs.smithi045.hxlkks 2024-04-16T11:51:15.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:15 smithi044 ceph-mon[93020]: pgmap v242: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:16.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:15 smithi045 ceph-mon[81423]: Upgrade: It appears safe to stop mds.cephfs.smithi045.hxlkks 2024-04-16T11:51:16.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:15 smithi045 ceph-mon[81423]: pgmap v242: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:17.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:17 smithi045 ceph-mon[81423]: Upgrade: Updating mds.cephfs.smithi045.hxlkks 2024-04-16T11:51:17.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:17.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:51:17.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:17 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:51:17.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:17 smithi045 ceph-mon[81423]: Deploying daemon mds.cephfs.smithi045.hxlkks on smithi045 2024-04-16T11:51:17.707 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:17 smithi045 ceph-mon[81423]: pgmap v243: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:17 smithi044 ceph-mon[93020]: Upgrade: Updating mds.cephfs.smithi045.hxlkks 2024-04-16T11:51:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hxlkks", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T11:51:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:17 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:51:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:17 smithi044 ceph-mon[93020]: Deploying daemon mds.cephfs.smithi045.hxlkks on smithi045 2024-04-16T11:51:17.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:17 smithi044 ceph-mon[93020]: pgmap v243: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:19.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 478, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127051 2024-04-16T11:51:19.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 590, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109593 2024-04-16T11:51:19.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 478, success, outsize: 120 2024-04-16T11:51:19.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 590, success, outsize: 120 2024-04-16T11:51:19.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.589+0000 7fdbf5200700 1 -- 172.21.15.44:0/1599852071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 msgr2=0x7fdbf0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:19.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.589+0000 7fdbf5200700 1 --2- 172.21.15.44:0/1599852071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0104b40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc004940 tx=0x7fdbdc009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:19.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.590+0000 7fdbf5200700 1 -- 172.21.15.44:0/1599852071 shutdown_connections 2024-04-16T11:51:19.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.590+0000 7fdbf5200700 1 --2- 172.21.15.44:0/1599852071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0104b40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:19.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.590+0000 7fdbf5200700 1 --2- 172.21.15.44:0/1599852071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbf0103420 0x7fdbf0103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:19.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.590+0000 7fdbf5200700 1 -- 172.21.15.44:0/1599852071 >> 172.21.15.44:0/1599852071 conn(0x7fdbf0072640 msgr2=0x7fdbf0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:19.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.590+0000 7fdbf5200700 1 -- 172.21.15.44:0/1599852071 shutdown_connections 2024-04-16T11:51:19.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.590+0000 7fdbf5200700 1 -- 172.21.15.44:0/1599852071 wait complete. 2024-04-16T11:51:19.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.591+0000 7fdbf5200700 1 Processor -- start 2024-04-16T11:51:19.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.591+0000 7fdbf5200700 1 -- start start 2024-04-16T11:51:19.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbf5200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbf0103420 0x7fdbf0113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:19.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbf5200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:19.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbf5200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf0114a90 con 0x7fdbf01046d0 2024-04-16T11:51:19.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbf5200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf0114c00 con 0x7fdbf0103420 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbed400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbed400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50976/0 (socket says 172.21.15.44:50976) 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbed400700 1 -- 172.21.15.44:0/2278771201 learned_addr learned my addr 172.21.15.44:0/2278771201 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.592+0000 7fdbede00700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbf0103420 0x7fdbf0113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbed400700 1 -- 172.21.15.44:0/2278771201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbf0103420 msgr2=0x7fdbf0113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbed400700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbf0103420 0x7fdbf0113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbed400700 1 -- 172.21.15.44:0/2278771201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc005040 con 0x7fdbf01046d0 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbed400700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0114490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b3b0 tx=0x7fdbdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbdc00e040 con 0x7fdbf01046d0 2024-04-16T11:51:19.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbdc004290 con 0x7fdbf01046d0 2024-04-16T11:51:19.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbdc01c630 con 0x7fdbf01046d0 2024-04-16T11:51:19.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.593+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf0119600 con 0x7fdbf01046d0 2024-04-16T11:51:19.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.594+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf0119bc0 con 0x7fdbf01046d0 2024-04-16T11:51:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.595+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf004c710 con 0x7fdbf01046d0 2024-04-16T11:51:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.596+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdbdc00bbf0 con 0x7fdbf01046d0 2024-04-16T11:51:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.597+0000 7fdbe6000700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdbd406ec00 0x7fdbd40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.597+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fdbdc08ffc0 con 0x7fdbf01046d0 2024-04-16T11:51:19.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.597+0000 7fdbede00700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdbd406ec00 0x7fdbd40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:19.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.598+0000 7fdbede00700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdbd406ec00 0x7fdbd40710b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0007fd0 tx=0x7fdbe0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:19.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.602+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdbdc0c4ff0 con 0x7fdbf01046d0 2024-04-16T11:51:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[93016]: 2024-04-16T11:51:19.474+0000 7f2289600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:51:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: pgmap v244: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:51:19.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:51:19.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: osdmap e91: 6 total, 6 up, 6 in 2024-04-16T11:51:19.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: Standby daemon mds.cephfs.smithi044.ufhhlc assigned to filesystem cephfs as rank 0 2024-04-16T11:51:19.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:51:19.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:51:19.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:19 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi044.ufhhlc=up:replay} 2 up:standby 2024-04-16T11:51:19.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: pgmap v244: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:19.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T11:51:19.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T11:51:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: osdmap e91: 6 total, 6 up, 6 in 2024-04-16T11:51:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: Standby daemon mds.cephfs.smithi044.ufhhlc assigned to filesystem cephfs as rank 0 2024-04-16T11:51:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T11:51:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T11:51:19.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:19 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi044.ufhhlc=up:replay} 2 up:standby 2024-04-16T11:51:19.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.856+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbf0108a10 con 0x7fdbd406ec00 2024-04-16T11:51:19.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.859+0000 7fdbe6000700 1 -- 172.21.15.44:0/2278771201 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdbf0108a10 con 0x7fdbd406ec00 2024-04-16T11:51:19.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdbd406ec00 msgr2=0x7fdbd40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:19.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdbd406ec00 0x7fdbd40710b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0007fd0 tx=0x7fdbe0009040 comp rx=0 tx=0).stop 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 msgr2=0x7fdbf0114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0114490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b3b0 tx=0x7fdbdc009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 shutdown_connections 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdbd406ec00 0x7fdbd40710b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbf0103420 0x7fdbf0113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 --2- 172.21.15.44:0/2278771201 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbf01046d0 0x7fdbf0114490 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.862+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 >> 172.21.15.44:0/2278771201 conn(0x7fdbf0072640 msgr2=0x7fdbf0073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.863+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 shutdown_connections 2024-04-16T11:51:19.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:19.863+0000 7fdbf5200700 1 -- 172.21.15.44:0/2278771201 wait complete. 2024-04-16T11:51:19.877 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:51:20.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 -- 172.21.15.44:0/1895458647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 msgr2=0x7f59d80fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 --2- 172.21.15.44:0/1895458647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d80fe460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f59c0004940 tx=0x7f59c0009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 -- 172.21.15.44:0/1895458647 shutdown_connections 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 --2- 172.21.15.44:0/1895458647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d80fe460 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 --2- 172.21.15.44:0/1895458647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59d80fce80 0x7f59d80fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 -- 172.21.15.44:0/1895458647 >> 172.21.15.44:0/1895458647 conn(0x7f59d80f8580 msgr2=0x7f59d80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 -- 172.21.15.44:0/1895458647 shutdown_connections 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.027+0000 7f59dca00700 1 -- 172.21.15.44:0/1895458647 wait complete. 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.028+0000 7f59dca00700 1 Processor -- start 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.028+0000 7f59dca00700 1 -- start start 2024-04-16T11:51:20.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59dca00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59d80fce80 0x7f59d810fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59dca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d8110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59dca00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8110670 con 0x7f59d80fdff0 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59dca00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d81107e0 con 0x7f59d80fce80 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59cd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d8110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59cd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d8110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50982/0 (socket says 172.21.15.44:50982) 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59cd400700 1 -- 172.21.15.44:0/204702966 learned_addr learned my addr 172.21.15.44:0/204702966 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59d5e00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59d80fce80 0x7f59d810fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59cd400700 1 -- 172.21.15.44:0/204702966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59d80fce80 msgr2=0x7f59d810fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59cd400700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59d80fce80 0x7f59d810fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.029+0000 7f59cd400700 1 -- 172.21.15.44:0/204702966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0005040 con 0x7f59d80fdff0 2024-04-16T11:51:20.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.030+0000 7f59cd400700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d8110070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59c000b3b0 tx=0x7f59c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.030+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59c000e040 con 0x7f59d80fdff0 2024-04-16T11:51:20.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.030+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59c0004290 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.030+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59c001c630 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.030+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d81151e0 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.030+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d81157a0 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.031+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8101980 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.032+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f59c000bbf0 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.033+0000 7f59cea00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59bc06ec00 0x7f59bc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.034+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f59c008fef0 con 0x7f59d80fdff0 2024-04-16T11:51:20.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.034+0000 7f59d5e00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59bc06ec00 0x7f59bc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.034+0000 7f59d5e00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59bc06ec00 0x7f59bc0710b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009fd0 tx=0x7f59c8009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:20.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.038+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f59c005c5d0 con 0x7f59d80fdff0 2024-04-16T11:51:20.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.295+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f59d802b2f0 con 0x7f59bc06ec00 2024-04-16T11:51:20.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.298+0000 7f59cea00700 1 -- 172.21.15.44:0/204702966 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f59d802b2f0 con 0x7f59bc06ec00 2024-04-16T11:51:20.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59bc06ec00 msgr2=0x7f59bc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59bc06ec00 0x7f59bc0710b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009fd0 tx=0x7f59c8009f20 comp rx=0 tx=0).stop 2024-04-16T11:51:20.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 msgr2=0x7f59d8110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d8110070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59c000b3b0 tx=0x7f59c0009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 shutdown_connections 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f59bc06ec00 0x7f59bc0710b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59d80fce80 0x7f59d810fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 --2- 172.21.15.44:0/204702966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f59d80fdff0 0x7f59d8110070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 >> 172.21.15.44:0/204702966 conn(0x7f59d80f8580 msgr2=0x7f59d80f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 shutdown_connections 2024-04-16T11:51:20.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.301+0000 7f59dca00700 1 -- 172.21.15.44:0/204702966 wait complete. 2024-04-16T11:51:20.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 -- 172.21.15.44:0/1219160766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4072830 msgr2=0x7fcbc4072c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 --2- 172.21.15.44:0/1219160766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4072830 0x7fcbc4072c40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4004940 tx=0x7fcbb4009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 -- 172.21.15.44:0/1219160766 shutdown_connections 2024-04-16T11:51:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 --2- 172.21.15.44:0/1219160766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbc4073180 0x7fcbc4073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 --2- 172.21.15.44:0/1219160766 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4072830 0x7fcbc4072c40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 -- 172.21.15.44:0/1219160766 >> 172.21.15.44:0/1219160766 conn(0x7fcbc40fa5d0 msgr2=0x7fcbc40fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 -- 172.21.15.44:0/1219160766 shutdown_connections 2024-04-16T11:51:20.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.465+0000 7fcbc3200700 1 -- 172.21.15.44:0/1219160766 wait complete. 2024-04-16T11:51:20.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.466+0000 7fcbc3200700 1 Processor -- start 2024-04-16T11:51:20.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.467+0000 7fcbc3200700 1 -- start start 2024-04-16T11:51:20.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.467+0000 7fcbc3200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbc4072830 0x7fcbc4113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.467+0000 7fcbc3200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 0x7fcbc4114440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.467+0000 7fcbc3200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4114a40 con 0x7fcbc4073180 2024-04-16T11:51:20.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.467+0000 7fcbc3200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4115290 con 0x7fcbc4072830 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 0x7fcbc4114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 0x7fcbc4114440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51008/0 (socket says 172.21.15.44:51008) 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 -- 172.21.15.44:0/3388953237 learned_addr learned my addr 172.21.15.44:0/3388953237 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbc1e00700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbc4072830 0x7fcbc4113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 -- 172.21.15.44:0/3388953237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbc4072830 msgr2=0x7fcbc4113f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbc4072830 0x7fcbc4113f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 -- 172.21.15.44:0/3388953237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4005040 con 0x7fcbc4073180 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.468+0000 7fcbbbe00700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 0x7fcbc4114440 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00c890 tx=0x7fcbac0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.469+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbac00a7d0 con 0x7fcbc4073180 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.469+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbac00ae00 con 0x7fcbc4073180 2024-04-16T11:51:20.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.469+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc4115570 con 0x7fcbc4073180 2024-04-16T11:51:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.469+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbac0113a0 con 0x7fcbc4073180 2024-04-16T11:51:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.469+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc4115ab0 con 0x7fcbc4073180 2024-04-16T11:51:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.470+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc405f5f0 con 0x7fcbc4073180 2024-04-16T11:51:20.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.471+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcbac012400 con 0x7fcbc4073180 2024-04-16T11:51:20.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.472+0000 7fcbba000700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcba806ec10 0x7fcba80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.472+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fcbac08ead0 con 0x7fcbc4073180 2024-04-16T11:51:20.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.473+0000 7fcbc1e00700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcba806ec10 0x7fcba80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.474+0000 7fcbc1e00700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcba806ec10 0x7fcba80710c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400b3b0 tx=0x7fcbb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:20.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.478+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcbac05b270 con 0x7fcbc4073180 2024-04-16T11:51:20.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 480, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127131 2024-04-16T11:51:20.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 480, success, outsize: 120 2024-04-16T11:51:20.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 592, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109594 2024-04-16T11:51:20.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 592, success, outsize: 120 2024-04-16T11:51:20.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.725+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbc4115e20 con 0x7fcba806ec10 2024-04-16T11:51:20.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.755+0000 7fcbba000700 1 -- 172.21.15.44:0/3388953237 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fcbc4115e20 con 0x7fcba806ec10 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (8m) 41s ago 18m 13.6M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (5m) 41s ago 19m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (5m) 14s ago 17m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (8m) 41s ago 18m 73.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (48s) 41s ago 11m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (81s) 41s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (11m) 14s ago 11m 17.0M - 17.2.7-847-g45317f95 b333581ccdf5 9b10c7482b3a 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (19s) 14s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (8m) 41s ago 20m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:51:20.754 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (7m) 14s ago 17m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (6m) 41s ago 20m 52.0M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (6m) 14s ago 17m 39.3M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (8m) 41s ago 19m 7708k - 1.5.0 0da6a335fe13 b259a1d3d74d 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (8m) 14s ago 17m 7684k - 1.5.0 0da6a335fe13 3a715e34eaa1 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (4m) 41s ago 16m 54.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (4m) 41s ago 15m 44.8M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (3m) 41s ago 15m 44.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (3m) 14s ago 14m 44.4M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (2m) 14s ago 14m 43.3M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (113s) 14s ago 13m 49.1M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:51:20.755 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (7m) 41s ago 18m 57.9M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.758+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcba806ec10 msgr2=0x7fcba80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.758+0000 7fcbc3200700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcba806ec10 0x7fcba80710c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400b3b0 tx=0x7fcbb4009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.758+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 msgr2=0x7fcbc4114440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.758+0000 7fcbc3200700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 0x7fcbc4114440 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00c890 tx=0x7fcbac0044c0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 shutdown_connections 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcba806ec10 0x7fcba80710c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbc4072830 0x7fcbc4113f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 --2- 172.21.15.44:0/3388953237 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcbc4073180 0x7fcbc4114440 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 >> 172.21.15.44:0/3388953237 conn(0x7fcbc40fa5d0 msgr2=0x7fcbc40fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 shutdown_connections 2024-04-16T11:51:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.759+0000 7fcbc3200700 1 -- 172.21.15.44:0/3388953237 wait complete. 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 -- 172.21.15.44:0/3980709903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d680ff020 msgr2=0x7f4d680ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 --2- 172.21.15.44:0/3980709903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d680ff020 0x7f4d680ff430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4d58004940 tx=0x7f4d58009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 -- 172.21.15.44:0/3980709903 shutdown_connections 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 --2- 172.21.15.44:0/3980709903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d681002d0 0x7f4d68100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 --2- 172.21.15.44:0/3980709903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d680ff020 0x7f4d680ff430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 -- 172.21.15.44:0/3980709903 >> 172.21.15.44:0/3980709903 conn(0x7f4d680fa950 msgr2=0x7f4d680fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 -- 172.21.15.44:0/3980709903 shutdown_connections 2024-04-16T11:51:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.919+0000 7f4d6e400700 1 -- 172.21.15.44:0/3980709903 wait complete. 2024-04-16T11:51:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.920+0000 7f4d6e400700 1 Processor -- start 2024-04-16T11:51:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.920+0000 7f4d6e400700 1 -- start start 2024-04-16T11:51:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d6e400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d680ff020 0x7f4d6810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d6e400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 0x7f4d681100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d6e400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681106a0 con 0x7f4d681002d0 2024-04-16T11:51:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d6e400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d68110810 con 0x7f4d680ff020 2024-04-16T11:51:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d66a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 0x7f4d681100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d66a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 0x7f4d681100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51028/0 (socket says 172.21.15.44:51028) 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d66a00700 1 -- 172.21.15.44:0/1094391871 learned_addr learned my addr 172.21.15.44:0/1094391871 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.921+0000 7f4d67400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d680ff020 0x7f4d6810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d66a00700 1 -- 172.21.15.44:0/1094391871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d680ff020 msgr2=0x7f4d6810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d66a00700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d680ff020 0x7f4d6810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d66a00700 1 -- 172.21.15.44:0/1094391871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58005040 con 0x7f4d681002d0 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d66a00700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 0x7f4d681100a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4d54002a50 tx=0x7f4d540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d54007cc0 con 0x7f4d681002d0 2024-04-16T11:51:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d5400f040 con 0x7f4d681002d0 2024-04-16T11:51:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d68115270 con 0x7f4d681002d0 2024-04-16T11:51:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d54009640 con 0x7f4d681002d0 2024-04-16T11:51:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.922+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d68115830 con 0x7f4d681002d0 2024-04-16T11:51:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.923+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6804c710 con 0x7f4d681002d0 2024-04-16T11:51:20.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.925+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4d540097a0 con 0x7f4d681002d0 2024-04-16T11:51:20.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.926+0000 7f4d5f400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4d4c06ec00 0x7f4d4c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:20.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.926+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f4d5408e980 con 0x7f4d681002d0 2024-04-16T11:51:20.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.926+0000 7f4d67400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4d4c06ec00 0x7f4d4c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:20.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.927+0000 7f4d67400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4d4c06ec00 0x7f4d4c0710b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800b3b0 tx=0x7f4d58009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:20.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:20.931+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4d5405af60 con 0x7f4d681002d0 2024-04-16T11:51:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:20 smithi044 ceph-mon[93020]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:20 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:reconnect 2024-04-16T11:51:20.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:20 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi044.ufhhlc=up:reconnect} 2 up:standby 2024-04-16T11:51:21.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:20 smithi045 ceph-mon[81423]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:21.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:20 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:reconnect 2024-04-16T11:51:21.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:20 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi044.ufhhlc=up:reconnect} 2 up:standby 2024-04-16T11:51:21.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.257+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4d68115a50 con 0x7f4d681002d0 2024-04-16T11:51:21.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.258+0000 7f4d5f400700 1 -- 172.21.15.44:0/1094391871 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4d68115a50 con 0x7f4d681002d0 2024-04-16T11:51:21.256 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:51:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:51:21.256 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:21.257 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:51:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 13 2024-04-16T11:51:21.258 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:51:21.258 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4d4c06ec00 msgr2=0x7f4d4c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4d4c06ec00 0x7f4d4c0710b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800b3b0 tx=0x7f4d58009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 msgr2=0x7f4d681100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 0x7f4d681100a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4d54002a50 tx=0x7f4d540046b0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 shutdown_connections 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f4d4c06ec00 0x7f4d4c0710b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d680ff020 0x7f4d6810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 --2- 172.21.15.44:0/1094391871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4d681002d0 0x7f4d681100a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 >> 172.21.15.44:0/1094391871 conn(0x7f4d680fa950 msgr2=0x7f4d680fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:21.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.261+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 shutdown_connections 2024-04-16T11:51:21.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.262+0000 7f4d6e400700 1 -- 172.21.15.44:0/1094391871 wait complete. 2024-04-16T11:51:21.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 -- 172.21.15.44:0/2739983226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec103420 msgr2=0x7f68ec103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2739983226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec103420 0x7f68ec103830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f68dc004940 tx=0x7f68dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 -- 172.21.15.44:0/2739983226 shutdown_connections 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2739983226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ec1046d0 0x7f68ec104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2739983226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec103420 0x7f68ec103830 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 -- 172.21.15.44:0/2739983226 >> 172.21.15.44:0/2739983226 conn(0x7f68ec072640 msgr2=0x7f68ec074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 -- 172.21.15.44:0/2739983226 shutdown_connections 2024-04-16T11:51:21.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.423+0000 7f68f0c00700 1 -- 172.21.15.44:0/2739983226 wait complete. 2024-04-16T11:51:21.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.424+0000 7f68f0c00700 1 Processor -- start 2024-04-16T11:51:21.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68f0c00700 1 -- start start 2024-04-16T11:51:21.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68f0c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ec1046d0 0x7f68ec114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:21.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68f0c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 0x7f68ec1195e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:21.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68f0c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec114a70 con 0x7f68ec1145b0 2024-04-16T11:51:21.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68f0c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec114bb0 con 0x7f68ec1046d0 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68e9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 0x7f68ec1195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68e9400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 0x7f68ec1195e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56892/0 (socket says 172.21.15.44:56892) 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.425+0000 7f68e9400700 1 -- 172.21.15.44:0/2723643326 learned_addr learned my addr 172.21.15.44:0/2723643326 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e9e00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ec1046d0 0x7f68ec114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e9400700 1 -- 172.21.15.44:0/2723643326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ec1046d0 msgr2=0x7f68ec114070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e9400700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ec1046d0 0x7f68ec114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e9400700 1 -- 172.21.15.44:0/2723643326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc005040 con 0x7f68ec1145b0 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e9400700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 0x7f68ec1195e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f68d800eb40 tx=0x7f68d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:21.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d8007cc0 con 0x7f68ec1145b0 2024-04-16T11:51:21.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68d800f040 con 0x7f68ec1145b0 2024-04-16T11:51:21.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d8009620 con 0x7f68ec1145b0 2024-04-16T11:51:21.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.426+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec119b20 con 0x7f68ec1145b0 2024-04-16T11:51:21.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.427+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec11a010 con 0x7f68ec1145b0 2024-04-16T11:51:21.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.428+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec04c710 con 0x7f68ec1145b0 2024-04-16T11:51:21.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.429+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f68d8009780 con 0x7f68ec1145b0 2024-04-16T11:51:21.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.430+0000 7f68e2000700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f68d006ec00 0x7f68d00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:21.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.430+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f68d808e790 con 0x7f68ec1145b0 2024-04-16T11:51:21.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.430+0000 7f68e9e00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f68d006ec00 0x7f68d00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:21.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.431+0000 7f68e9e00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f68d006ec00 0x7f68d00710b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68dc013740 tx=0x7f68dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:21.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.435+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f68d805aea0 con 0x7f68ec1145b0 2024-04-16T11:51:21.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.716+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f68ec10bf10 con 0x7f68ec1145b0 2024-04-16T11:51:21.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.720+0000 7f68e2000700 1 -- 172.21.15.44:0/2723643326 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1746 (secure 0 0 0) 0x7f68d805aa30 con 0x7f68ec1145b0 2024-04-16T11:51:21.719 INFO:teuthology.orchestra.run.smithi044.stdout:e35 2024-04-16T11:51:21.720 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:51:21.720 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:51:21.720 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:51:21.720 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:21.720 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:51:21.721 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:51:21.721 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 35 2024-04-16T11:51:21.721 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:51:21.721 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:51:21.721 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:51:20.497626+0000 2024-04-16T11:51:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:51:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:51:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:51:21.722 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 91 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:51:21.723 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=34432} 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 0 members: 2024-04-16T11:51:21.724 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:34432} state up:rejoin seq 23 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:21.725 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:51:21.725 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:21.725 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:21.725 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:44319} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f68d006ec00 msgr2=0x7f68d00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f68d006ec00 0x7f68d00710b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68dc013740 tx=0x7f68dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 msgr2=0x7f68ec1195e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 0x7f68ec1195e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f68d800eb40 tx=0x7f68d800b040 comp rx=0 tx=0).stop 2024-04-16T11:51:21.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 shutdown_connections 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f68d006ec00 0x7f68d00710b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ec1046d0 0x7f68ec114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 --2- 172.21.15.44:0/2723643326 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ec1145b0 0x7f68ec1195e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 >> 172.21.15.44:0/2723643326 conn(0x7f68ec072640 msgr2=0x7f68ec073db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 shutdown_connections 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.724+0000 7f68f0c00700 1 -- 172.21.15.44:0/2723643326 wait complete. 2024-04-16T11:51:21.727 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 35 2024-04-16T11:51:21.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 -- 172.21.15.44:0/779555046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fcc20 msgr2=0x7f9d200fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 --2- 172.21.15.44:0/779555046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fcc20 0x7f9d200fd030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9d14004940 tx=0x7f9d14009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:21.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 -- 172.21.15.44:0/779555046 shutdown_connections 2024-04-16T11:51:21.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 --2- 172.21.15.44:0/779555046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200fded0 0x7f9d200fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 --2- 172.21.15.44:0/779555046 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fcc20 0x7f9d200fd030 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 -- 172.21.15.44:0/779555046 >> 172.21.15.44:0/779555046 conn(0x7f9d200f83a0 msgr2=0x7f9d200fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 -- 172.21.15.44:0/779555046 shutdown_connections 2024-04-16T11:51:21.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.876+0000 7f9d26c00700 1 -- 172.21.15.44:0/779555046 wait complete. 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.877+0000 7f9d26c00700 1 Processor -- start 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.877+0000 7f9d26c00700 1 -- start start 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d26c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200fcc20 0x7f9d2010d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d26c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 0x7f9d2010dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d26c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2010e390 con 0x7f9d200fded0 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d26c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2010ebe0 con 0x7f9d200fcc20 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d24e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 0x7f9d2010dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d24e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 0x7f9d2010dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56900/0 (socket says 172.21.15.44:56900) 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d24e00700 1 -- 172.21.15.44:0/885529270 learned_addr learned my addr 172.21.15.44:0/885529270 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.878+0000 7f9d25800700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200fcc20 0x7f9d2010d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d24e00700 1 -- 172.21.15.44:0/885529270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200fcc20 msgr2=0x7f9d2010d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d24e00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200fcc20 0x7f9d2010d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d24e00700 1 -- 172.21.15.44:0/885529270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14005040 con 0x7f9d200fded0 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d24e00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 0x7f9d2010dd90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00c8c0 tx=0x7f9d0c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d0c00a980 con 0x7f9d200fded0 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d0c00ce60 con 0x7f9d200fded0 2024-04-16T11:51:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d0c0115e0 con 0x7f9d200fded0 2024-04-16T11:51:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2010eec0 con 0x7f9d200fded0 2024-04-16T11:51:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.879+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2010f400 con 0x7f9d200fded0 2024-04-16T11:51:21.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.880+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d200fc020 con 0x7f9d200fded0 2024-04-16T11:51:21.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.882+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9d0c012400 con 0x7f9d200fded0 2024-04-16T11:51:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.883+0000 7f9d1d600700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9d0806ec00 0x7f9d080710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.883+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f9d0c08eb90 con 0x7f9d200fded0 2024-04-16T11:51:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.883+0000 7f9d25800700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9d0806ec00 0x7f9d080710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.884+0000 7f9d25800700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9d0806ec00 0x7f9d080710b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400b3b0 tx=0x7f9d14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:21.888+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9d0c05b270 con 0x7f9d200fded0 2024-04-16T11:51:21.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00199997 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00299995 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: pgmap v246: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1094391871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:rejoin 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: fsmap cephfs:1/1 {0=cephfs.smithi044.ufhhlc=up:rejoin} 2 up:standby 2024-04-16T11:51:21.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:21 smithi044 ceph-mon[93020]: daemon mds.cephfs.smithi044.ufhhlc is now active in filesystem cephfs as rank 0 2024-04-16T11:51:22.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:22.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: reconnect by client.14520 192.168.0.1:0/3632682047 after 0.00199997 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: reconnect by client.24317 192.168.0.1:0/3169505875 after 0.00299995 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: pgmap v246: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1094391871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:rejoin 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: fsmap cephfs:1/1 {0=cephfs.smithi044.ufhhlc=up:rejoin} 2 up:standby 2024-04-16T11:51:22.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:21 smithi045 ceph-mon[81423]: daemon mds.cephfs.smithi044.ufhhlc is now active in filesystem cephfs as rank 0 2024-04-16T11:51:22.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.129+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d2010f840 con 0x7f9d0806ec00 2024-04-16T11:51:22.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.131+0000 7f9d1d600700 1 -- 172.21.15.44:0/885529270 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9d2010f840 con 0x7f9d0806ec00 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "crash" 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "15/21 daemons upgraded", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T11:51:22.129 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:51:22.130 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:51:22.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.133+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9d0806ec00 msgr2=0x7f9d080710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.133+0000 7f9d26c00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9d0806ec00 0x7f9d080710b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400b3b0 tx=0x7f9d14009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.133+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 msgr2=0x7f9d2010dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.133+0000 7f9d26c00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 0x7f9d2010dd90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00c8c0 tx=0x7f9d0c0044c0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 shutdown_connections 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9d0806ec00 0x7f9d080710b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d200fcc20 0x7f9d2010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 --2- 172.21.15.44:0/885529270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d200fded0 0x7f9d2010dd90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 >> 172.21.15.44:0/885529270 conn(0x7f9d200f83a0 msgr2=0x7f9d200f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 shutdown_connections 2024-04-16T11:51:22.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.134+0000 7f9d26c00700 1 -- 172.21.15.44:0/885529270 wait complete. 2024-04-16T11:51:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 -- 172.21.15.44:0/3137642543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c0fef00 msgr2=0x7f785c0ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 --2- 172.21.15.44:0/3137642543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c0fef00 0x7f785c0ff310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f784c004940 tx=0x7f784c009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 -- 172.21.15.44:0/3137642543 shutdown_connections 2024-04-16T11:51:22.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 --2- 172.21.15.44:0/3137642543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f785c100070 0x7f785c1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 --2- 172.21.15.44:0/3137642543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c0fef00 0x7f785c0ff310 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 -- 172.21.15.44:0/3137642543 >> 172.21.15.44:0/3137642543 conn(0x7f785c0fa790 msgr2=0x7f785c0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:22.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 -- 172.21.15.44:0/3137642543 shutdown_connections 2024-04-16T11:51:22.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.277+0000 7f7861200700 1 -- 172.21.15.44:0/3137642543 wait complete. 2024-04-16T11:51:22.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.278+0000 7f7861200700 1 Processor -- start 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f7861200700 1 -- start start 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f7861200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f785c0fef00 0x7f785c10fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f7861200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 0x7f785c10ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f7861200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1105a0 con 0x7f785c100070 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f7861200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c110df0 con 0x7f785c0fef00 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f785aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 0x7f785c10ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f785aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 0x7f785c10ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56926/0 (socket says 172.21.15.44:56926) 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f785aa00700 1 -- 172.21.15.44:0/2777407172 learned_addr learned my addr 172.21.15.44:0/2777407172 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.279+0000 7f785b400700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f785c0fef00 0x7f785c10fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f785aa00700 1 -- 172.21.15.44:0/2777407172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f785c0fef00 msgr2=0x7f785c10fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:22.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f785aa00700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f785c0fef00 0x7f785c10fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f785aa00700 1 -- 172.21.15.44:0/2777407172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c005040 con 0x7f785c100070 2024-04-16T11:51:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f785aa00700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 0x7f785c10ffa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f784800c830 tx=0x7f7848004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f784800a760 con 0x7f785c100070 2024-04-16T11:51:22.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f784800ad90 con 0x7f785c100070 2024-04-16T11:51:22.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78480113a0 con 0x7f785c100070 2024-04-16T11:51:22.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.280+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c1110d0 con 0x7f785c100070 2024-04-16T11:51:22.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.281+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c111610 con 0x7f785c100070 2024-04-16T11:51:22.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.282+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c0fe290 con 0x7f785c100070 2024-04-16T11:51:22.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.283+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7848012400 con 0x7f785c100070 2024-04-16T11:51:22.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.284+0000 7f7853400700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f784006ec00 0x7f78400710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:22.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.284+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f784808eb10 con 0x7f785c100070 2024-04-16T11:51:22.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.284+0000 7f785b400700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f784006ec00 0x7f78400710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:22.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.285+0000 7f785b400700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f784006ec00 0x7f78400710b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f784c004f90 tx=0x7f784c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:22.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.289+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f784805b1f0 con 0x7f785c100070 2024-04-16T11:51:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.604+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f785c04bed0 con 0x7f785c100070 2024-04-16T11:51:22.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.605+0000 7f7853400700 1 -- 172.21.15.44:0/2777407172 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f784805ad80 con 0x7f785c100070 2024-04-16T11:51:22.603 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:51:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f784006ec00 msgr2=0x7f78400710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f784006ec00 0x7f78400710b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f784c004f90 tx=0x7f784c009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 msgr2=0x7f785c10ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 0x7f785c10ffa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f784800c830 tx=0x7f7848004470 comp rx=0 tx=0).stop 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 shutdown_connections 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f784006ec00 0x7f78400710b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f785c0fef00 0x7f785c10fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 --2- 172.21.15.44:0/2777407172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f785c100070 0x7f785c10ffa0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 >> 172.21.15.44:0/2777407172 conn(0x7f785c0fa790 msgr2=0x7f785c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 shutdown_connections 2024-04-16T11:51:22.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:22.608+0000 7f7861200700 1 -- 172.21.15.44:0/2777407172 wait complete. 2024-04-16T11:51:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:22 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2723643326' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:51:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:22 smithi044 ceph-mon[93020]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:51:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:22 smithi044 ceph-mon[93020]: Cluster is now healthy 2024-04-16T11:51:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:22 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:active 2024-04-16T11:51:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:22 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2 up:standby 2024-04-16T11:51:22.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:22 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2777407172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:51:23.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:22 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2723643326' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:51:23.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:22 smithi045 ceph-mon[81423]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T11:51:23.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:22 smithi045 ceph-mon[81423]: Cluster is now healthy 2024-04-16T11:51:23.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:22 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] up:active 2024-04-16T11:51:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:22 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 2 up:standby 2024-04-16T11:51:23.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:22 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2777407172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:51:23.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:23 smithi044 ceph-mon[93020]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:23.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:23 smithi044 ceph-mon[93020]: pgmap v247: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-16T11:51:24.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:23 smithi045 ceph-mon[81423]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:24.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:23 smithi045 ceph-mon[81423]: pgmap v247: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-16T11:51:25.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:25 smithi044 ceph-mon[93020]: pgmap v248: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1 op/s 2024-04-16T11:51:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:26.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:25 smithi045 ceph-mon[81423]: pgmap v248: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1 op/s 2024-04-16T11:51:26.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:28.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:27 smithi045 ceph-mon[81423]: pgmap v249: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T11:51:28.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:27 smithi044 ceph-mon[93020]: pgmap v249: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: pgmap v250: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: mds.? [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] up:boot 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:51:29.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:51:29.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: pgmap v250: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T11:51:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: mds.? [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] up:boot 2024-04-16T11:51:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: fsmap cephfs:1 {0=cephfs.smithi044.ufhhlc=up:active} 3 up:standby 2024-04-16T11:51:29.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:51:31.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:31 smithi044 ceph-mon[93020]: pgmap v251: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 7.8 KiB/s rd, 183 B/s wr, 7 op/s 2024-04-16T11:51:32.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:31 smithi045 ceph-mon[81423]: pgmap v251: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 7.8 KiB/s rd, 183 B/s wr, 7 op/s 2024-04-16T11:51:33.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:33.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:33 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:33.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:33 smithi045 ceph-mon[81423]: pgmap v252: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T11:51:33.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:33 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:33.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:33 smithi044 ceph-mon[93020]: pgmap v252: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T11:51:35.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:35.803 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:35.803 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:35 smithi045 ceph-mon[81423]: pgmap v253: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T11:51:35.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:35.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:35 smithi044 ceph-mon[93020]: pgmap v253: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T11:51:37.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:37 smithi044 ceph-mon[93020]: pgmap v254: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T11:51:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:37 smithi045 ceph-mon[81423]: pgmap v254: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-16T11:51:38.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 594, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110579 2024-04-16T11:51:38.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 594, success, outsize: 120 2024-04-16T11:51:38.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 596, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110662 2024-04-16T11:51:38.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 596, success, outsize: 120 2024-04-16T11:51:38.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 598, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110748 2024-04-16T11:51:38.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 598, success, outsize: 120 2024-04-16T11:51:38.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 600, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110840 2024-04-16T11:51:38.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 600, success, outsize: 120 2024-04-16T11:51:38.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 602, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110929 2024-04-16T11:51:38.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 602, success, outsize: 120 2024-04-16T11:51:38.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 604, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111012 2024-04-16T11:51:38.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 604, success, outsize: 120 2024-04-16T11:51:38.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 606, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111098 2024-04-16T11:51:38.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 606, success, outsize: 120 2024-04-16T11:51:38.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 608, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111190 2024-04-16T11:51:38.644 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 608, success, outsize: 120 2024-04-16T11:51:38.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 610, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111279 2024-04-16T11:51:38.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 610, success, outsize: 120 2024-04-16T11:51:38.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 612, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111362 2024-04-16T11:51:38.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 612, success, outsize: 120 2024-04-16T11:51:38.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 614, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111448 2024-04-16T11:51:38.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 614, success, outsize: 120 2024-04-16T11:51:38.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 616, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111540 2024-04-16T11:51:38.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 616, success, outsize: 120 2024-04-16T11:51:38.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 618, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111629 2024-04-16T11:51:38.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 618, success, outsize: 120 2024-04-16T11:51:38.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 620, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111712 2024-04-16T11:51:38.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 620, success, outsize: 120 2024-04-16T11:51:38.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 622, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111798 2024-04-16T11:51:38.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 622, success, outsize: 120 2024-04-16T11:51:38.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 624, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111890 2024-04-16T11:51:38.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 624, success, outsize: 120 2024-04-16T11:51:38.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 626, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111979 2024-04-16T11:51:38.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 626, success, outsize: 120 2024-04-16T11:51:38.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 628, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112062 2024-04-16T11:51:38.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 628, success, outsize: 120 2024-04-16T11:51:38.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 630, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112148 2024-04-16T11:51:38.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 630, success, outsize: 120 2024-04-16T11:51:38.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 632, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112240 2024-04-16T11:51:38.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 632, success, outsize: 120 2024-04-16T11:51:38.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 634, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112329 2024-04-16T11:51:38.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 634, success, outsize: 120 2024-04-16T11:51:38.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 636, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112412 2024-04-16T11:51:38.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 636, success, outsize: 120 2024-04-16T11:51:39.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 638, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112498 2024-04-16T11:51:39.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 638, success, outsize: 120 2024-04-16T11:51:39.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 640, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112590 2024-04-16T11:51:39.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 640, success, outsize: 120 2024-04-16T11:51:39.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 642, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112679 2024-04-16T11:51:39.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 642, success, outsize: 120 2024-04-16T11:51:39.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 644, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112762 2024-04-16T11:51:39.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 644, success, outsize: 120 2024-04-16T11:51:39.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 646, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112848 2024-04-16T11:51:39.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 646, success, outsize: 120 2024-04-16T11:51:39.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 648, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112940 2024-04-16T11:51:39.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 648, success, outsize: 120 2024-04-16T11:51:39.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:39 smithi045 ceph-mon[81423]: pgmap v255: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:39.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:39 smithi044 ceph-mon[93020]: pgmap v255: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.hlafhv"}]': finished 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.ufhhlc"}]': finished 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.hxlkks"}]': finished 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.noofsd"}]': finished 2024-04-16T11:51:41.228 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.229 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.hlafhv"}]: dispatch 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.hlafhv"}]': finished 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.ufhhlc"}]: dispatch 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi044.ufhhlc"}]': finished 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.hxlkks"}]: dispatch 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.hxlkks"}]': finished 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.noofsd"}]: dispatch 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi045.noofsd"}]': finished 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.332 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:41.333 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:42.048 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all mds 2024-04-16T11:51:42.048 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all rgw 2024-04-16T11:51:42.048 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T11:51:42.048 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T11:51:42.049 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all iscsi 2024-04-16T11:51:42.049 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all nfs 2024-04-16T11:51:42.049 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: Upgrade: Setting container_image for all nvmeof 2024-04-16T11:51:42.049 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:41 smithi044 ceph-mon[93020]: pgmap v256: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:42.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all mds 2024-04-16T11:51:42.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all rgw 2024-04-16T11:51:42.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T11:51:42.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T11:51:42.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all iscsi 2024-04-16T11:51:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all nfs 2024-04-16T11:51:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: Upgrade: Setting container_image for all nvmeof 2024-04-16T11:51:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:41 smithi045 ceph-mon[81423]: pgmap v256: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:44.127 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:43 smithi044 ceph-mon[93020]: pgmap v257: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:44.128 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:43 smithi044 ceph-mon[93020]: Upgrade: Updating node-exporter.smithi044 (1/2) 2024-04-16T11:51:44.128 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:43 smithi044 ceph-mon[93020]: Deploying daemon node-exporter.smithi044 on smithi044 2024-04-16T11:51:44.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:43 smithi045 ceph-mon[81423]: pgmap v257: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:44.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:43 smithi045 ceph-mon[81423]: Upgrade: Updating node-exporter.smithi044 (1/2) 2024-04-16T11:51:44.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:43 smithi045 ceph-mon[81423]: Deploying daemon node-exporter.smithi044 on smithi044 2024-04-16T11:51:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:45 smithi044 ceph-mon[93020]: pgmap v258: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:46.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:45 smithi045 ceph-mon[81423]: pgmap v258: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:47 smithi044 ceph-mon[93020]: pgmap v259: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:47 smithi045 ceph-mon[81423]: pgmap v259: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:50.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:49 smithi044 ceph-mon[93020]: pgmap v260: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:50.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:49 smithi045 ceph-mon[81423]: pgmap v260: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:50 smithi045 ceph-mon[81423]: Upgrade: Updating node-exporter.smithi045 (2/2) 2024-04-16T11:51:51.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:50 smithi045 ceph-mon[81423]: Deploying daemon node-exporter.smithi045 on smithi045 2024-04-16T11:51:51.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:50 smithi044 ceph-mon[93020]: Upgrade: Updating node-exporter.smithi045 (2/2) 2024-04-16T11:51:51.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:50 smithi044 ceph-mon[93020]: Deploying daemon node-exporter.smithi045 on smithi045 2024-04-16T11:51:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:51 smithi045 ceph-mon[81423]: pgmap v261: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:52.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:51 smithi044 ceph-mon[93020]: pgmap v261: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.782+0000 7f87c9c00700 1 -- 172.21.15.44:0/480549131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c40ff000 msgr2=0x7f87c40ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.782+0000 7f87c9c00700 1 --2- 172.21.15.44:0/480549131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c40ff000 0x7f87c40ff410 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87b4004940 tx=0x7f87b4009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.783+0000 7f87c9c00700 1 -- 172.21.15.44:0/480549131 shutdown_connections 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.783+0000 7f87c9c00700 1 --2- 172.21.15.44:0/480549131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c41002b0 0x7f87c4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.783+0000 7f87c9c00700 1 --2- 172.21.15.44:0/480549131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c40ff000 0x7f87c40ff410 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.783+0000 7f87c9c00700 1 -- 172.21.15.44:0/480549131 >> 172.21.15.44:0/480549131 conn(0x7f87c40fa950 msgr2=0x7f87c40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:52.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.783+0000 7f87c9c00700 1 -- 172.21.15.44:0/480549131 shutdown_connections 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.783+0000 7f87c9c00700 1 -- 172.21.15.44:0/480549131 wait complete. 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.784+0000 7f87c9c00700 1 Processor -- start 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.784+0000 7f87c9c00700 1 -- start start 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c9c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c40ff000 0x7f87c410fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c9c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 0x7f87c41100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c9c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c41106d0 con 0x7f87c41002b0 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c9c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4110f20 con 0x7f87c40ff000 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c2000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 0x7f87c41100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c2000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 0x7f87c41100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44352/0 (socket says 172.21.15.44:44352) 2024-04-16T11:51:52.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c2000700 1 -- 172.21.15.44:0/362459757 learned_addr learned my addr 172.21.15.44:0/362459757 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:52.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.785+0000 7f87c2a00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c40ff000 0x7f87c410fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:52.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.786+0000 7f87c2000700 1 -- 172.21.15.44:0/362459757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c40ff000 msgr2=0x7f87c410fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:52.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.786+0000 7f87c2000700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c40ff000 0x7f87c410fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:52.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.786+0000 7f87c2000700 1 -- 172.21.15.44:0/362459757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b4005040 con 0x7f87c41002b0 2024-04-16T11:51:52.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.786+0000 7f87c2000700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 0x7f87c41100d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f87b0002a80 tx=0x7f87b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.786+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b0007cc0 con 0x7f87c41002b0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.787+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87b000f040 con 0x7f87c41002b0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.787+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b000dac0 con 0x7f87c41002b0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.787+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c4111200 con 0x7f87c41002b0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.787+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c4111740 con 0x7f87c41002b0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.788+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c405f5f0 con 0x7f87c41002b0 2024-04-16T11:51:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.789+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87b000dc20 con 0x7f87c41002b0 2024-04-16T11:51:52.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.790+0000 7f87baa00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87a806ec10 0x7f87a80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:52.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.790+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f87b008e7d0 con 0x7f87c41002b0 2024-04-16T11:51:52.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.790+0000 7f87c2a00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87a806ec10 0x7f87a80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:52.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.791+0000 7f87c2a00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87a806ec10 0x7f87a80710c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b3b0 tx=0x7f87b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:52.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:52.795+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f87b005adb0 con 0x7f87c41002b0 2024-04-16T11:51:53.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.050+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87c4111ae0 con 0x7f87a806ec10 2024-04-16T11:51:53.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.053+0000 7f87baa00700 1 -- 172.21.15.44:0/362459757 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f87c4111ae0 con 0x7f87a806ec10 2024-04-16T11:51:53.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87a806ec10 msgr2=0x7f87a80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87a806ec10 0x7f87a80710c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b3b0 tx=0x7f87b4009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:53.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 msgr2=0x7f87c41100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 0x7f87c41100d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f87b0002a80 tx=0x7f87b00046b0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 shutdown_connections 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f87a806ec10 0x7f87a80710c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87c40ff000 0x7f87c410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 --2- 172.21.15.44:0/362459757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f87c41002b0 0x7f87c41100d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 >> 172.21.15.44:0/362459757 conn(0x7f87c40fa950 msgr2=0x7f87c40fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 shutdown_connections 2024-04-16T11:51:53.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.056+0000 7f87c9c00700 1 -- 172.21.15.44:0/362459757 wait complete. 2024-04-16T11:51:53.068 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.214+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1881780064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0102320 msgr2=0x7f6fa0104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.214+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1881780064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0102320 0x7f6fa0104700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c004940 tx=0x7f6f8c009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.215+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1881780064 shutdown_connections 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.215+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1881780064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0102320 0x7f6fa0104700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.215+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1881780064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa0066f60 0x7f6fa0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.215+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1881780064 >> 172.21.15.44:0/1881780064 conn(0x7f6fa00f7450 msgr2=0x7f6fa00f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.215+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1881780064 shutdown_connections 2024-04-16T11:51:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.215+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1881780064 wait complete. 2024-04-16T11:51:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.216+0000 7f6fa7a00700 1 Processor -- start 2024-04-16T11:51:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.216+0000 7f6fa7a00700 1 -- start start 2024-04-16T11:51:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa7a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 0x7f6fa010fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:53.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa7a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa0102320 0x7f6fa01103d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa7a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa01109d0 con 0x7f6fa0066f60 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa7a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa0110b40 con 0x7f6fa0102320 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 0x7f6fa010fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 0x7f6fa010fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44376/0 (socket says 172.21.15.44:44376) 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6fa5000700 1 -- 172.21.15.44:0/1108688436 learned_addr learned my addr 172.21.15.44:0/1108688436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.217+0000 7f6f9fe00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa0102320 0x7f6fa01103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6fa5000700 1 -- 172.21.15.44:0/1108688436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa0102320 msgr2=0x7f6fa01103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6fa5000700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa0102320 0x7f6fa01103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6fa5000700 1 -- 172.21.15.44:0/1108688436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c005040 con 0x7f6fa0066f60 2024-04-16T11:51:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6fa5000700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 0x7f6fa010fe90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6f94002a50 tx=0x7f6f94004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f94007cc0 con 0x7f6fa0066f60 2024-04-16T11:51:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f9400f040 con 0x7f6fa0066f60 2024-04-16T11:51:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.218+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa01155a0 con 0x7f6fa0066f60 2024-04-16T11:51:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.219+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9400d9c0 con 0x7f6fa0066f60 2024-04-16T11:51:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.219+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa0115b60 con 0x7f6fa0066f60 2024-04-16T11:51:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.220+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa004c710 con 0x7f6fa0066f60 2024-04-16T11:51:53.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.221+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6f94009730 con 0x7f6fa0066f60 2024-04-16T11:51:53.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.222+0000 7f6f9cc00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f6f8806ec00 0x7f6f880710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.222+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f6f9408e840 con 0x7f6fa0066f60 2024-04-16T11:51:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.222+0000 7f6f9fe00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f6f8806ec00 0x7f6f880710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.223+0000 7f6f9fe00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f6f8806ec00 0x7f6f880710b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00b3b0 tx=0x7f6f8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.227+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6f9405ae20 con 0x7f6fa0066f60 2024-04-16T11:51:53.287 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:53 smithi045 ceph-mon[81423]: pgmap v262: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:53.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:53 smithi044 ceph-mon[93020]: pgmap v262: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:53.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.483+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fa0103f30 con 0x7f6f8806ec00 2024-04-16T11:51:53.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.486+0000 7f6f9cc00700 1 -- 172.21.15.44:0/1108688436 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f6fa0103f30 con 0x7f6f8806ec00 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.488+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f6f8806ec00 msgr2=0x7f6f880710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.488+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f6f8806ec00 0x7f6f880710b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00b3b0 tx=0x7f6f8c009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.488+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 msgr2=0x7f6fa010fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.488+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 0x7f6fa010fe90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6f94002a50 tx=0x7f6f94004600 comp rx=0 tx=0).stop 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 shutdown_connections 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f6f8806ec00 0x7f6f880710b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fa0066f60 0x7f6fa010fe90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 --2- 172.21.15.44:0/1108688436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa0102320 0x7f6fa01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 >> 172.21.15.44:0/1108688436 conn(0x7f6fa00f7450 msgr2=0x7f6fa00f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 shutdown_connections 2024-04-16T11:51:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.489+0000 7f6fa7a00700 1 -- 172.21.15.44:0/1108688436 wait complete. 2024-04-16T11:51:53.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.645+0000 7fdc85e00700 1 -- 172.21.15.44:0/3367701464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 msgr2=0x7fdc80104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.645+0000 7fdc85e00700 1 --2- 172.21.15.44:0/3367701464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80104b40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdc68004940 tx=0x7fdc68009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.645+0000 7fdc85e00700 1 -- 172.21.15.44:0/3367701464 shutdown_connections 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.645+0000 7fdc85e00700 1 --2- 172.21.15.44:0/3367701464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80104b40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.645+0000 7fdc85e00700 1 --2- 172.21.15.44:0/3367701464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 0x7fdc80103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.645+0000 7fdc85e00700 1 -- 172.21.15.44:0/3367701464 >> 172.21.15.44:0/3367701464 conn(0x7fdc80072640 msgr2=0x7fdc80074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.646+0000 7fdc85e00700 1 -- 172.21.15.44:0/3367701464 shutdown_connections 2024-04-16T11:51:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.646+0000 7fdc85e00700 1 -- 172.21.15.44:0/3367701464 wait complete. 2024-04-16T11:51:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.647+0000 7fdc85e00700 1 Processor -- start 2024-04-16T11:51:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.647+0000 7fdc85e00700 1 -- start start 2024-04-16T11:51:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.647+0000 7fdc85e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 0x7fdc80113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.647+0000 7fdc85e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.647+0000 7fdc85e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc80114a90 con 0x7fdc801046d0 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.647+0000 7fdc85e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc80114c00 con 0x7fdc80103420 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc76000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc76000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44392/0 (socket says 172.21.15.44:44392) 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc76000700 1 -- 172.21.15.44:0/394987930 learned_addr learned my addr 172.21.15.44:0/394987930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc7ea00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 0x7fdc80113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc76000700 1 -- 172.21.15.44:0/394987930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 msgr2=0x7fdc80113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc76000700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 0x7fdc80113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc76000700 1 -- 172.21.15.44:0/394987930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc68005040 con 0x7fdc801046d0 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.648+0000 7fdc7ea00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 0x7fdc80113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:51:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.649+0000 7fdc76000700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80114490 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009150 tx=0x7fdc68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:53.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.649+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc6800e040 con 0x7fdc801046d0 2024-04-16T11:51:53.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.649+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc6800bcc0 con 0x7fdc801046d0 2024-04-16T11:51:53.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.649+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc6801c630 con 0x7fdc801046d0 2024-04-16T11:51:53.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.649+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc80119600 con 0x7fdc801046d0 2024-04-16T11:51:53.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.649+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc80119bc0 con 0x7fdc801046d0 2024-04-16T11:51:53.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.650+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8004c710 con 0x7fdc801046d0 2024-04-16T11:51:53.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.652+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdc68013420 con 0x7fdc801046d0 2024-04-16T11:51:53.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.653+0000 7fdc77400700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc6406ec00 0x7fdc640710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:53.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.653+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fdc6808f770 con 0x7fdc801046d0 2024-04-16T11:51:53.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.653+0000 7fdc7ea00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc6406ec00 0x7fdc640710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:53.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.654+0000 7fdc7ea00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc6406ec00 0x7fdc640710b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009fd0 tx=0x7fdc70009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:53.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.658+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdc6805be50 con 0x7fdc801046d0 2024-04-16T11:51:53.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.908+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc80108a10 con 0x7fdc6406ec00 2024-04-16T11:51:53.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.923+0000 7fdc77400700 1 -- 172.21.15.44:0/394987930 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fdc80108a10 con 0x7fdc6406ec00 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (9m) 74s ago 19m 13.6M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (5m) 74s ago 19m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (5m) 21s ago 18m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (8m) 74s ago 19m 73.5M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (81s) 74s ago 12m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (114s) 74s ago 12m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (25s) 21s ago 12m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 e08aedf9cf6c 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (52s) 21s ago 12m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (8m) 74s ago 21m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (7m) 21s ago 18m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (7m) 74s ago 21m 52.0M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:51:53.920 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (6m) 21s ago 18m 40.1M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 starting - - - - 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 starting - - - - 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (5m) 74s ago 17m 54.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (4m) 74s ago 16m 44.8M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (4m) 74s ago 15m 44.1M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (3m) 21s ago 15m 44.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (3m) 21s ago 14m 42.5M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (2m) 21s ago 14m 49.5M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:51:53.921 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (7m) 74s ago 18m 57.9M - 2.43.0 a07b618ecd1d fd9366f19728 2024-04-16T11:51:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc6406ec00 msgr2=0x7fdc640710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc6406ec00 0x7fdc640710b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009fd0 tx=0x7fdc70009ee0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 msgr2=0x7fdc80114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80114490 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009150 tx=0x7fdc68009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 shutdown_connections 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc6406ec00 0x7fdc640710b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc80103420 0x7fdc80113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 --2- 172.21.15.44:0/394987930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc801046d0 0x7fdc80114490 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.925+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 >> 172.21.15.44:0/394987930 conn(0x7fdc80072640 msgr2=0x7fdc80073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.926+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 shutdown_connections 2024-04-16T11:51:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:53.926+0000 7fdc85e00700 1 -- 172.21.15.44:0/394987930 wait complete. 2024-04-16T11:51:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 -- 172.21.15.44:0/3904502410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 msgr2=0x7fdc74070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 --2- 172.21.15.44:0/3904502410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74070460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdc64004940 tx=0x7fdc64009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 -- 172.21.15.44:0/3904502410 shutdown_connections 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 --2- 172.21.15.44:0/3904502410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc740709a0 0x7fdc74070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 --2- 172.21.15.44:0/3904502410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74070460 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 -- 172.21.15.44:0/3904502410 >> 172.21.15.44:0/3904502410 conn(0x7fdc740f8530 msgr2=0x7fdc740fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 -- 172.21.15.44:0/3904502410 shutdown_connections 2024-04-16T11:51:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.077+0000 7fdc78c00700 1 -- 172.21.15.44:0/3904502410 wait complete. 2024-04-16T11:51:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.078+0000 7fdc78c00700 1 Processor -- start 2024-04-16T11:51:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.078+0000 7fdc78c00700 1 -- start start 2024-04-16T11:51:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.078+0000 7fdc78c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc740709a0 0x7fdc74114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.078+0000 7fdc78c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.078+0000 7fdc78c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc74114b50 con 0x7fdc74071f50 2024-04-16T11:51:54.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.078+0000 7fdc78c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc741153a0 con 0x7fdc740709a0 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74114550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44408/0 (socket says 172.21.15.44:44408) 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71400700 1 -- 172.21.15.44:0/494128408 learned_addr learned my addr 172.21.15.44:0/494128408 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71e00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc740709a0 0x7fdc74114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71400700 1 -- 172.21.15.44:0/494128408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc740709a0 msgr2=0x7fdc74114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71400700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc740709a0 0x7fdc74114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.079+0000 7fdc71400700 1 -- 172.21.15.44:0/494128408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc64005040 con 0x7fdc74071f50 2024-04-16T11:51:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.080+0000 7fdc71400700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74114550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000eb40 tx=0x7fdc6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.080+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc60007cc0 con 0x7fdc74071f50 2024-04-16T11:51:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.080+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc6000f040 con 0x7fdc74071f50 2024-04-16T11:51:54.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.080+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc74115620 con 0x7fdc74071f50 2024-04-16T11:51:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.080+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc6000ca10 con 0x7fdc74071f50 2024-04-16T11:51:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.080+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc74115b60 con 0x7fdc74071f50 2024-04-16T11:51:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.081+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7405f5f0 con 0x7fdc74071f50 2024-04-16T11:51:54.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.084+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdc6000cb70 con 0x7fdc74071f50 2024-04-16T11:51:54.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.085+0000 7fdc6a000700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc5806ec10 0x7fdc580710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:54.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.085+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fdc6008e670 con 0x7fdc74071f50 2024-04-16T11:51:54.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.085+0000 7fdc71e00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc5806ec10 0x7fdc580710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:54.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.086+0000 7fdc71e00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc5806ec10 0x7fdc580710c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc64013870 tx=0x7fdc64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:54.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.090+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdc6005ad80 con 0x7fdc74071f50 2024-04-16T11:51:54.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.420+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdc7410bf90 con 0x7fdc74071f50 2024-04-16T11:51:54.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.420+0000 7fdc6a000700 1 -- 172.21.15.44:0/494128408 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fdc6005a910 con 0x7fdc74071f50 2024-04-16T11:51:54.417 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:51:54.417 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T11:51:54.418 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:51:54.419 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc5806ec10 msgr2=0x7fdc580710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc5806ec10 0x7fdc580710c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc64013870 tx=0x7fdc64009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 msgr2=0x7fdc74114550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74114550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000eb40 tx=0x7fdc6000b040 comp rx=0 tx=0).stop 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 shutdown_connections 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdc5806ec10 0x7fdc580710c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc740709a0 0x7fdc74114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 --2- 172.21.15.44:0/494128408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc74071f50 0x7fdc74114550 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 >> 172.21.15.44:0/494128408 conn(0x7fdc740f8530 msgr2=0x7fdc740f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:54.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 shutdown_connections 2024-04-16T11:51:54.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.424+0000 7fdc78c00700 1 -- 172.21.15.44:0/494128408 wait complete. 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 -- 172.21.15.44:0/132174044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680fff70 msgr2=0x7ff368102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 --2- 172.21.15.44:0/132174044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680fff70 0x7ff368102390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff354004940 tx=0x7ff354009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 -- 172.21.15.44:0/132174044 shutdown_connections 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 --2- 172.21.15.44:0/132174044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680fff70 0x7ff368102390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 --2- 172.21.15.44:0/132174044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3680662f0 0x7ff3680ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 -- 172.21.15.44:0/132174044 >> 172.21.15.44:0/132174044 conn(0x7ff3680f74c0 msgr2=0x7ff3680f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:54.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 -- 172.21.15.44:0/132174044 shutdown_connections 2024-04-16T11:51:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.577+0000 7ff36ec00700 1 -- 172.21.15.44:0/132174044 wait complete. 2024-04-16T11:51:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.578+0000 7ff36ec00700 1 Processor -- start 2024-04-16T11:51:54.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff36ec00700 1 -- start start 2024-04-16T11:51:54.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff36ec00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 0x7ff3680fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff36ec00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3680fff70 0x7ff3680fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff36ec00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680fbee0 con 0x7ff3680662f0 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff36ec00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680fc050 con 0x7ff3680fff70 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff367e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 0x7ff3680fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff367e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 0x7ff3680fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44420/0 (socket says 172.21.15.44:44420) 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff367e00700 1 -- 172.21.15.44:0/2796007824 learned_addr learned my addr 172.21.15.44:0/2796007824 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.579+0000 7ff367400700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3680fff70 0x7ff3680fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff367e00700 1 -- 172.21.15.44:0/2796007824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3680fff70 msgr2=0x7ff3680fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff367e00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3680fff70 0x7ff3680fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff367e00700 1 -- 172.21.15.44:0/2796007824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354005040 con 0x7ff3680662f0 2024-04-16T11:51:54.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff367e00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 0x7ff3680fd460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff35800eb40 tx=0x7ff35800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff358007cc0 con 0x7ff3680662f0 2024-04-16T11:51:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff35800f040 con 0x7ff3680662f0 2024-04-16T11:51:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.580+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3680fc2d0 con 0x7ff3680662f0 2024-04-16T11:51:54.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.581+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff368115a00 con 0x7ff3680662f0 2024-04-16T11:51:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.581+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff35800c9d0 con 0x7ff3680662f0 2024-04-16T11:51:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.582+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36805f5f0 con 0x7ff3680662f0 2024-04-16T11:51:54.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.583+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff35800cb30 con 0x7ff3680662f0 2024-04-16T11:51:54.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.584+0000 7ff35fe00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff34c06ec10 0x7ff34c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.584+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7ff35808e420 con 0x7ff3680662f0 2024-04-16T11:51:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.584+0000 7ff367400700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff34c06ec10 0x7ff34c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:54.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.585+0000 7ff367400700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff34c06ec10 0x7ff34c0710c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff35400b0d0 tx=0x7ff354009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:54.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.589+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff3580570a0 con 0x7ff3680662f0 2024-04-16T11:51:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.883+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff36804bed0 con 0x7ff3680662f0 2024-04-16T11:51:54.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.884+0000 7ff35fe00700 1 -- 172.21.15.44:0/2796007824 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 37 v37) v1 ==== 76+0+1932 (secure 0 0 0) 0x7ff35805a760 con 0x7ff3680662f0 2024-04-16T11:51:54.881 INFO:teuthology.orchestra.run.smithi044.stdout:e37 2024-04-16T11:51:54.881 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:51:54.881 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:51:54.881 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:51:54.881 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 36 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:51:22.505053+0000 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:51:54.882 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 91 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=34432} 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:51:54.883 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 34432 members: 34432 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:34432} state up:active seq 24 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:54.884 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:54.885 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:51:54.885 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:51:54.885 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:54.885 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:44319} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:54.885 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:44341} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.886+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff34c06ec10 msgr2=0x7ff34c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.886+0000 7ff36ec00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff34c06ec10 0x7ff34c0710c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff35400b0d0 tx=0x7ff354009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.886+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 msgr2=0x7ff3680fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.886+0000 7ff36ec00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 0x7ff3680fd460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff35800eb40 tx=0x7ff35800b040 comp rx=0 tx=0).stop 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 shutdown_connections 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff34c06ec10 0x7ff34c0710c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3680662f0 0x7ff3680fd460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 --2- 172.21.15.44:0/2796007824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3680fff70 0x7ff3680fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 >> 172.21.15.44:0/2796007824 conn(0x7ff3680f74c0 msgr2=0x7ff3680f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 shutdown_connections 2024-04-16T11:51:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:54.887+0000 7ff36ec00700 1 -- 172.21.15.44:0/2796007824 wait complete. 2024-04-16T11:51:54.887 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 -- 172.21.15.44:0/2923007599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d8066290 msgr2=0x7f03d80ff860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 --2- 172.21.15.44:0/2923007599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d8066290 0x7f03d80ff860 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009ce0 tx=0x7f03c8004d10 comp rx=0 tx=0).stop 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 -- 172.21.15.44:0/2923007599 shutdown_connections 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 --2- 172.21.15.44:0/2923007599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d80ffda0 0x7f03d8102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 --2- 172.21.15.44:0/2923007599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d8066290 0x7f03d80ff860 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 -- 172.21.15.44:0/2923007599 >> 172.21.15.44:0/2923007599 conn(0x7f03d80f7270 msgr2=0x7f03d80f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:55.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 -- 172.21.15.44:0/2923007599 shutdown_connections 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:55.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:54 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/494128408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.035+0000 7f03d7e00700 1 -- 172.21.15.44:0/2923007599 wait complete. 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.036+0000 7f03d7e00700 1 Processor -- start 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.036+0000 7f03d7e00700 1 -- start start 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d7e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d8066290 0x7f03d810fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d7e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d80ffda0 0x7f03d8110320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d7e00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8110920 con 0x7f03d80ffda0 2024-04-16T11:51:55.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d7e00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8111010 con 0x7f03d8066290 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d8066290 0x7f03d810fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d6000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d80ffda0 0x7f03d8110320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d6a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d8066290 0x7f03d810fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.44:44200/0 (socket says 172.21.15.44:44200) 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.037+0000 7f03d6a00700 1 -- 172.21.15.44:0/712295059 learned_addr learned my addr 172.21.15.44:0/712295059 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.038+0000 7f03d6000700 1 -- 172.21.15.44:0/712295059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d8066290 msgr2=0x7f03d810fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.038+0000 7f03d6000700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d8066290 0x7f03d810fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.038+0000 7f03d6000700 1 -- 172.21.15.44:0/712295059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8009940 con 0x7f03d80ffda0 2024-04-16T11:51:55.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.039+0000 7f03d6000700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d80ffda0 0x7f03d8110320 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f03c4002870 tx=0x7f03c4004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:55.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.039+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c4007cc0 con 0x7f03d80ffda0 2024-04-16T11:51:55.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.039+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d81112f0 con 0x7f03d80ffda0 2024-04-16T11:51:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.039+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d81118b0 con 0x7f03d80ffda0 2024-04-16T11:51:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.039+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03c400f040 con 0x7f03d80ffda0 2024-04-16T11:51:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.039+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c4013980 con 0x7f03d80ffda0 2024-04-16T11:51:55.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.042+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b80052b0 con 0x7f03d80ffda0 2024-04-16T11:51:55.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.043+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f03c401c400 con 0x7f03d80ffda0 2024-04-16T11:51:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.044+0000 7f03cea00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03bc06ec10 0x7f03bc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.044+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f03c408e600 con 0x7f03d80ffda0 2024-04-16T11:51:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.044+0000 7f03d6a00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03bc06ec10 0x7f03bc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:55.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.045+0000 7f03d6a00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03bc06ec10 0x7f03bc0710c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03c800b740 tx=0x7f03c8004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:55.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.050+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f03c405ac60 con 0x7f03d80ffda0 2024-04-16T11:51:55.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:51:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:55.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:54 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/494128408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:51:55.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.297+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03b8000bc0 con 0x7f03bc06ec10 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.299+0000 7f03cea00700 1 -- 172.21.15.44:0/712295059 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f03b8000bc0 con 0x7f03bc06ec10 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:51:55.296 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "mds" 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:51:55.297 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03bc06ec10 msgr2=0x7f03bc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03bc06ec10 0x7f03bc0710c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03c800b740 tx=0x7f03c8004d10 comp rx=0 tx=0).stop 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d80ffda0 msgr2=0x7f03d8110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d80ffda0 0x7f03d8110320 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f03c4002870 tx=0x7f03c4004570 comp rx=0 tx=0).stop 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 shutdown_connections 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f03bc06ec10 0x7f03bc0710c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03d8066290 0x7f03d810fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 --2- 172.21.15.44:0/712295059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f03d80ffda0 0x7f03d8110320 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 >> 172.21.15.44:0/712295059 conn(0x7f03d80f7270 msgr2=0x7f03d80fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 shutdown_connections 2024-04-16T11:51:55.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.302+0000 7f03d7e00700 1 -- 172.21.15.44:0/712295059 wait complete. 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 -- 172.21.15.44:0/1384202051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 msgr2=0x7fa008100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 --2- 172.21.15.44:0/1384202051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008100740 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4004940 tx=0x7f9ff4009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 -- 172.21.15.44:0/1384202051 shutdown_connections 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 --2- 172.21.15.44:0/1384202051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008100740 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 --2- 172.21.15.44:0/1384202051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0080ff020 0x7fa0080ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 -- 172.21.15.44:0/1384202051 >> 172.21.15.44:0/1384202051 conn(0x7fa0080fa950 msgr2=0x7fa0080fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 -- 172.21.15.44:0/1384202051 shutdown_connections 2024-04-16T11:51:55.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.444+0000 7fa00e800700 1 -- 172.21.15.44:0/1384202051 wait complete. 2024-04-16T11:51:55.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.445+0000 7fa00e800700 1 Processor -- start 2024-04-16T11:51:55.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.445+0000 7fa00e800700 1 -- start start 2024-04-16T11:51:55.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.445+0000 7fa00e800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0080ff020 0x7fa00810fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:55.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.445+0000 7fa00e800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.445+0000 7fa00e800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008110670 con 0x7fa0081002d0 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.445+0000 7fa00e800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0081107e0 con 0x7fa0080ff020 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44446/0 (socket says 172.21.15.44:44446) 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 -- 172.21.15.44:0/944127369 learned_addr learned my addr 172.21.15.44:0/944127369 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 -- 172.21.15.44:0/944127369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0080ff020 msgr2=0x7fa00810fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0080ff020 0x7fa00810fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 -- 172.21.15.44:0/944127369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4005040 con 0x7fa0081002d0 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.446+0000 7fa006a00700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008110070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b3b0 tx=0x7f9ff4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.447+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ff400e040 con 0x7fa0081002d0 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.447+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ff4004290 con 0x7fa0081002d0 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.447+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ff401c630 con 0x7fa0081002d0 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.447+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081151e0 con 0x7fa0081002d0 2024-04-16T11:51:55.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.447+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0081157a0 con 0x7fa0081002d0 2024-04-16T11:51:55.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.448+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00804c710 con 0x7fa0081002d0 2024-04-16T11:51:55.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.449+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ff400bbf0 con 0x7fa0081002d0 2024-04-16T11:51:55.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.450+0000 7f9fff400700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9fec06ec00 0x7f9fec0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:51:55.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.450+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f9ff408fef0 con 0x7fa0081002d0 2024-04-16T11:51:55.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.450+0000 7fa007400700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9fec06ec00 0x7f9fec0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:51:55.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.451+0000 7fa007400700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9fec06ec00 0x7f9fec0710b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8004d00 tx=0x7f9ff8009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:51:55.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.455+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9ff405c5d0 con 0x7fa0081002d0 2024-04-16T11:51:55.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.801+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa008107b10 con 0x7fa0081002d0 2024-04-16T11:51:55.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.801+0000 7f9fff400700 1 -- 172.21.15.44:0/944127369 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f9ff405c160 con 0x7fa0081002d0 2024-04-16T11:51:55.798 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.805+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9fec06ec00 msgr2=0x7f9fec0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.805+0000 7fa00e800700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9fec06ec00 0x7f9fec0710b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8004d00 tx=0x7f9ff8009ce0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.805+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 msgr2=0x7fa008110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.805+0000 7fa00e800700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008110070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b3b0 tx=0x7f9ff4009e30 comp rx=0 tx=0).stop 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 shutdown_connections 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9fec06ec00 0x7f9fec0710b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0080ff020 0x7fa00810fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 --2- 172.21.15.44:0/944127369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0081002d0 0x7fa008110070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 >> 172.21.15.44:0/944127369 conn(0x7fa0080fa950 msgr2=0x7fa0080fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 shutdown_connections 2024-04-16T11:51:55.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:51:55.806+0000 7fa00e800700 1 -- 172.21.15.44:0/944127369 wait complete. 2024-04-16T11:51:56.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:55 smithi045 ceph-mon[81423]: pgmap v263: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:56.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:55 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2796007824' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:51:56.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:56.117 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:55 smithi044 ceph-mon[93020]: pgmap v263: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:56.117 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:55 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2796007824' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:51:56.117 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:51:57.044 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:56 smithi044 ceph-mon[93020]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:57.045 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:56 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/944127369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:51:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:56 smithi045 ceph-mon[81423]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:51:57.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:56 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/944127369' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:51:58.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:57 smithi045 ceph-mon[81423]: pgmap v264: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:58.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:58.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:57 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:57 smithi044 ceph-mon[93020]: pgmap v264: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:51:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:51:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:57 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:00.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:59 smithi045 ceph-mon[81423]: pgmap v265: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:00.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:00.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:51:59 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:00.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:59 smithi044 ceph-mon[93020]: pgmap v265: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:00.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:51:59 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:01 smithi045 ceph-mon[81423]: pgmap v266: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:01.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:01 smithi044 ceph-mon[93020]: pgmap v266: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:03.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 650, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114299 2024-04-16T11:52:03.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 650, success, outsize: 120 2024-04-16T11:52:03.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 652, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114393 2024-04-16T11:52:03.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 652, success, outsize: 120 2024-04-16T11:52:03.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 654, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114490 2024-04-16T11:52:03.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 654, success, outsize: 120 2024-04-16T11:52:03.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 656, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114593 2024-04-16T11:52:03.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 656, success, outsize: 120 2024-04-16T11:52:03.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 658, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114693 2024-04-16T11:52:03.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 658, success, outsize: 120 2024-04-16T11:52:03.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 660, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114787 2024-04-16T11:52:03.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 660, success, outsize: 120 2024-04-16T11:52:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 662, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114884 2024-04-16T11:52:03.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 662, success, outsize: 120 2024-04-16T11:52:03.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:03 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:03.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:03 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:03.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:03 smithi045 ceph-mon[81423]: pgmap v267: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:03.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 664, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114987 2024-04-16T11:52:03.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 664, success, outsize: 120 2024-04-16T11:52:03.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 666, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115087 2024-04-16T11:52:03.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 666, success, outsize: 120 2024-04-16T11:52:03.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 668, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115181 2024-04-16T11:52:03.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 668, success, outsize: 120 2024-04-16T11:52:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 670, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115278 2024-04-16T11:52:03.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 670, success, outsize: 120 2024-04-16T11:52:03.676 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:03 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:03.676 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:03 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:03.676 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:03 smithi044 ceph-mon[93020]: pgmap v267: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:03.683 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 672, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115381 2024-04-16T11:52:03.683 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 672, success, outsize: 120 2024-04-16T11:52:03.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 674, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115481 2024-04-16T11:52:03.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 674, success, outsize: 120 2024-04-16T11:52:03.723 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 676, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115575 2024-04-16T11:52:03.723 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 676, success, outsize: 120 2024-04-16T11:52:03.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 678, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115672 2024-04-16T11:52:03.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 678, success, outsize: 120 2024-04-16T11:52:03.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 680, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115775 2024-04-16T11:52:03.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 680, success, outsize: 120 2024-04-16T11:52:03.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 682, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115875 2024-04-16T11:52:03.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 682, success, outsize: 120 2024-04-16T11:52:03.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 684, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115970 2024-04-16T11:52:03.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 684, success, outsize: 120 2024-04-16T11:52:03.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 686, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116067 2024-04-16T11:52:03.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 686, success, outsize: 120 2024-04-16T11:52:03.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 688, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116170 2024-04-16T11:52:03.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 688, success, outsize: 120 2024-04-16T11:52:03.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 690, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116270 2024-04-16T11:52:03.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 690, success, outsize: 120 2024-04-16T11:52:03.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 692, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116364 2024-04-16T11:52:03.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 692, success, outsize: 120 2024-04-16T11:52:03.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 694, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116461 2024-04-16T11:52:03.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 694, success, outsize: 120 2024-04-16T11:52:03.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 696, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116564 2024-04-16T11:52:03.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 696, success, outsize: 120 2024-04-16T11:52:03.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 698, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116664 2024-04-16T11:52:03.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 698, success, outsize: 120 2024-04-16T11:52:03.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 700, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116758 2024-04-16T11:52:03.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 700, success, outsize: 120 2024-04-16T11:52:03.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 702, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116855 2024-04-16T11:52:03.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 702, success, outsize: 120 2024-04-16T11:52:03.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 704, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116958 2024-04-16T11:52:03.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr: unique: 704, success, outsize: 120 2024-04-16T11:52:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:05 smithi044 ceph-mon[93020]: pgmap v268: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:06.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:06.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:05 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:05 smithi045 ceph-mon[81423]: pgmap v268: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:05 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:06.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 482, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128902 2024-04-16T11:52:06.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 482, success, outsize: 120 2024-04-16T11:52:06.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 484, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128984 2024-04-16T11:52:06.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 484, success, outsize: 120 2024-04-16T11:52:06.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 486, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129069 2024-04-16T11:52:06.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 486, success, outsize: 120 2024-04-16T11:52:06.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 488, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129160 2024-04-16T11:52:06.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 488, success, outsize: 120 2024-04-16T11:52:06.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 490, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129248 2024-04-16T11:52:06.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 490, success, outsize: 120 2024-04-16T11:52:06.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 492, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129330 2024-04-16T11:52:06.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 492, success, outsize: 120 2024-04-16T11:52:06.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 494, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129415 2024-04-16T11:52:06.796 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 494, success, outsize: 120 2024-04-16T11:52:06.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 496, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129506 2024-04-16T11:52:06.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 496, success, outsize: 120 2024-04-16T11:52:06.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 498, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129594 2024-04-16T11:52:06.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 498, success, outsize: 120 2024-04-16T11:52:06.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 500, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129676 2024-04-16T11:52:06.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 500, success, outsize: 120 2024-04-16T11:52:06.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 502, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129761 2024-04-16T11:52:06.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 502, success, outsize: 120 2024-04-16T11:52:06.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 504, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129852 2024-04-16T11:52:06.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 504, success, outsize: 120 2024-04-16T11:52:06.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 506, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129940 2024-04-16T11:52:06.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 506, success, outsize: 120 2024-04-16T11:52:06.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 508, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130022 2024-04-16T11:52:06.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 508, success, outsize: 120 2024-04-16T11:52:06.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 510, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130107 2024-04-16T11:52:06.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 510, success, outsize: 120 2024-04-16T11:52:07.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 512, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130198 2024-04-16T11:52:07.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 512, success, outsize: 120 2024-04-16T11:52:07.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 514, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130286 2024-04-16T11:52:07.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 514, success, outsize: 120 2024-04-16T11:52:07.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 516, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130368 2024-04-16T11:52:07.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 516, success, outsize: 120 2024-04-16T11:52:07.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 518, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130453 2024-04-16T11:52:07.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 518, success, outsize: 120 2024-04-16T11:52:07.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 520, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130544 2024-04-16T11:52:07.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 520, success, outsize: 120 2024-04-16T11:52:07.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 522, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130632 2024-04-16T11:52:07.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 522, success, outsize: 120 2024-04-16T11:52:07.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 524, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130714 2024-04-16T11:52:07.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 524, success, outsize: 120 2024-04-16T11:52:07.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 526, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130799 2024-04-16T11:52:07.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 526, success, outsize: 120 2024-04-16T11:52:07.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 528, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130890 2024-04-16T11:52:07.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 528, success, outsize: 120 2024-04-16T11:52:07.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 530, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130978 2024-04-16T11:52:07.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 530, success, outsize: 120 2024-04-16T11:52:07.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 532, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131060 2024-04-16T11:52:07.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 532, success, outsize: 120 2024-04-16T11:52:07.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 534, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131145 2024-04-16T11:52:07.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 534, success, outsize: 120 2024-04-16T11:52:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 536, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131236 2024-04-16T11:52:07.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 536, success, outsize: 120 2024-04-16T11:52:07.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:07 smithi044 ceph-mon[93020]: pgmap v269: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:08.131 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:07 smithi045 ceph-mon[81423]: pgmap v269: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:09.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:09.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:09.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:52:09.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: pgmap v270: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.581 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:09 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: pgmap v270: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.691 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.692 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.692 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.692 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.692 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:09.692 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:09 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:10.637 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:10.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:11.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:11 smithi044 ceph-mon[93020]: pgmap v271: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:11.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:11 smithi045 ceph-mon[81423]: pgmap v271: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:12.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:12 smithi044 ceph-mon[93020]: Upgrade: Updating prometheus.smithi044 2024-04-16T11:52:12.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:12 smithi044 ceph-mon[93020]: Deploying daemon prometheus.smithi044 on smithi044 2024-04-16T11:52:12.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:12 smithi045 ceph-mon[81423]: Upgrade: Updating prometheus.smithi044 2024-04-16T11:52:12.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:12 smithi045 ceph-mon[81423]: Deploying daemon prometheus.smithi044 on smithi044 2024-04-16T11:52:13.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:13 smithi044 ceph-mon[93020]: pgmap v272: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:13.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:13 smithi045 ceph-mon[81423]: pgmap v272: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:15.926 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:15 smithi044 ceph-mon[93020]: pgmap v273: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:16.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:15 smithi045 ceph-mon[81423]: pgmap v273: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:17.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:17 smithi044 ceph-mon[93020]: pgmap v274: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:18.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:17 smithi045 ceph-mon[81423]: pgmap v274: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:20.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:19 smithi045 ceph-mon[81423]: pgmap v275: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:20.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:20.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:20.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:20.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:19 smithi044 ceph-mon[93020]: pgmap v275: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:20.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:20.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:20.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:21.713 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:21.714 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:21 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:21.714 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:21 smithi044 ceph-mon[93020]: pgmap v276: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:21.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:21.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:21 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:21.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:21 smithi045 ceph-mon[81423]: pgmap v276: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:23.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:23 smithi044 ceph-mon[93020]: pgmap v277: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:24.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:23 smithi045 ceph-mon[81423]: pgmap v277: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:25.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 -- 172.21.15.44:0/2562831167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2324103420 msgr2=0x7f2324103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:25.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 --2- 172.21.15.44:0/2562831167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2324103420 0x7f2324103830 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2318004940 tx=0x7f2318009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:25.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 -- 172.21.15.44:0/2562831167 shutdown_connections 2024-04-16T11:52:25.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 --2- 172.21.15.44:0/2562831167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23241046d0 0x7f2324104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:25.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 --2- 172.21.15.44:0/2562831167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2324103420 0x7f2324103830 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:25.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 -- 172.21.15.44:0/2562831167 >> 172.21.15.44:0/2562831167 conn(0x7f2324072640 msgr2=0x7f2324074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:25.965 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:25 smithi044 ceph-mon[93020]: pgmap v278: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:25.965 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 -- 172.21.15.44:0/2562831167 shutdown_connections 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.967+0000 7f232bc00700 1 -- 172.21.15.44:0/2562831167 wait complete. 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.968+0000 7f232bc00700 1 Processor -- start 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.968+0000 7f232bc00700 1 -- start start 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.968+0000 7f232bc00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2324103420 0x7f2324113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.968+0000 7f232bc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 0x7f23241144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.968+0000 7f232bc00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324114aa0 con 0x7f23241046d0 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.968+0000 7f232bc00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324114c10 con 0x7f2324103420 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2323e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 0x7f23241144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2323e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 0x7f23241144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38496/0 (socket says 172.21.15.44:38496) 2024-04-16T11:52:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2323e00700 1 -- 172.21.15.44:0/2445717755 learned_addr learned my addr 172.21.15.44:0/2445717755 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2329200700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2324103420 0x7f2324113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2323e00700 1 -- 172.21.15.44:0/2445717755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2324103420 msgr2=0x7f2324113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2323e00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2324103420 0x7f2324113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.969+0000 7f2323e00700 1 -- 172.21.15.44:0/2445717755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318005040 con 0x7f23241046d0 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.970+0000 7f2323e00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 0x7f23241144a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f231000c8c0 tx=0x7f23100044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.970+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231000a980 con 0x7f23241046d0 2024-04-16T11:52:25.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.970+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f231000ce60 con 0x7f23241046d0 2024-04-16T11:52:25.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.970+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23100115e0 con 0x7f23241046d0 2024-04-16T11:52:25.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.970+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2324119670 con 0x7f23241046d0 2024-04-16T11:52:25.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.970+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2324119c30 con 0x7f23241046d0 2024-04-16T11:52:25.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.971+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232404c710 con 0x7f23241046d0 2024-04-16T11:52:25.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.973+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2310012400 con 0x7f23241046d0 2024-04-16T11:52:25.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.974+0000 7f2320c00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f230c06ec00 0x7f230c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:25.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.974+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f231008eb90 con 0x7f23241046d0 2024-04-16T11:52:25.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.974+0000 7f2329200700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f230c06ec00 0x7f230c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:25.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.975+0000 7f2329200700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f230c06ec00 0x7f230c0710b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f231800b3b0 tx=0x7f2318009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:25.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:25.979+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f231005b270 con 0x7f23241046d0 2024-04-16T11:52:26.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:25 smithi045 ceph-mon[81423]: pgmap v278: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:26.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:26.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.228+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2324108a60 con 0x7f230c06ec00 2024-04-16T11:52:26.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.232+0000 7f2320c00700 1 -- 172.21.15.44:0/2445717755 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7f2324108a60 con 0x7f230c06ec00 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.234+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f230c06ec00 msgr2=0x7f230c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.234+0000 7f232bc00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f230c06ec00 0x7f230c0710b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f231800b3b0 tx=0x7f2318009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 msgr2=0x7f23241144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 0x7f23241144a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f231000c8c0 tx=0x7f23100044c0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 shutdown_connections 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f230c06ec00 0x7f230c0710b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2324103420 0x7f2324113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 --2- 172.21.15.44:0/2445717755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f23241046d0 0x7f23241144a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 >> 172.21.15.44:0/2445717755 conn(0x7f2324072640 msgr2=0x7f2324073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 shutdown_connections 2024-04-16T11:52:26.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.235+0000 7f232bc00700 1 -- 172.21.15.44:0/2445717755 wait complete. 2024-04-16T11:52:26.251 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.397+0000 7fe5d5000700 1 -- 172.21.15.44:0/3894157747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 msgr2=0x7fe5d0102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.397+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3894157747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d0102180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc004940 tx=0x7fe5bc009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.397+0000 7fe5d5000700 1 -- 172.21.15.44:0/3894157747 shutdown_connections 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.397+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3894157747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d0102180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.397+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3894157747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d0066ce0 0x7fe5d00ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.397+0000 7fe5d5000700 1 -- 172.21.15.44:0/3894157747 >> 172.21.15.44:0/3894157747 conn(0x7fe5d00f7290 msgr2=0x7fe5d00f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.398+0000 7fe5d5000700 1 -- 172.21.15.44:0/3894157747 shutdown_connections 2024-04-16T11:52:26.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.398+0000 7fe5d5000700 1 -- 172.21.15.44:0/3894157747 wait complete. 2024-04-16T11:52:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.398+0000 7fe5d5000700 1 Processor -- start 2024-04-16T11:52:26.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5d5000700 1 -- start start 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5d5000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d0066ce0 0x7fe5d010fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5d5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d010ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5d5000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d0110640 con 0x7fe5d00ffd60 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5d5000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d0110e30 con 0x7fe5d0066ce0 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d010ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5cea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d010ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38524/0 (socket says 172.21.15.44:38524) 2024-04-16T11:52:26.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.399+0000 7fe5cea00700 1 -- 172.21.15.44:0/3792756458 learned_addr learned my addr 172.21.15.44:0/3792756458 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:26.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.400+0000 7fe5cf400700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d0066ce0 0x7fe5d010fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:26.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.400+0000 7fe5cea00700 1 -- 172.21.15.44:0/3792756458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d0066ce0 msgr2=0x7fe5d010fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.400+0000 7fe5cea00700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d0066ce0 0x7fe5d010fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.400+0000 7fe5cea00700 1 -- 172.21.15.44:0/3792756458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc005040 con 0x7fe5d00ffd60 2024-04-16T11:52:26.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.400+0000 7fe5cea00700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d010ffb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00b3b0 tx=0x7fe5bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:26.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.401+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5bc00e040 con 0x7fe5d00ffd60 2024-04-16T11:52:26.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.401+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5bc004290 con 0x7fe5d00ffd60 2024-04-16T11:52:26.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.401+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d01110b0 con 0x7fe5d00ffd60 2024-04-16T11:52:26.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.401+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5bc01c630 con 0x7fe5d00ffd60 2024-04-16T11:52:26.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.401+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d0111670 con 0x7fe5d00ffd60 2024-04-16T11:52:26.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.402+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d00faf30 con 0x7fe5d00ffd60 2024-04-16T11:52:26.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.403+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe5bc00bbf0 con 0x7fe5d00ffd60 2024-04-16T11:52:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.404+0000 7fe5c7400700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5b406ec00 0x7fe5b40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.404+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fe5bc08ffc0 con 0x7fe5d00ffd60 2024-04-16T11:52:26.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.405+0000 7fe5cf400700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5b406ec00 0x7fe5b40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:26.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.405+0000 7fe5cf400700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5b406ec00 0x7fe5b40710b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0006fd0 tx=0x7fe5c0008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:26.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.411+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe5bc0c4ff0 con 0x7fe5d00ffd60 2024-04-16T11:52:26.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.683+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5d0111950 con 0x7fe5b406ec00 2024-04-16T11:52:26.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.686+0000 7fe5c7400700 1 -- 172.21.15.44:0/3792756458 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7fe5d0111950 con 0x7fe5b406ec00 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5b406ec00 msgr2=0x7fe5b40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5b406ec00 0x7fe5b40710b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0006fd0 tx=0x7fe5c0008040 comp rx=0 tx=0).stop 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 msgr2=0x7fe5d010ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d010ffb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00b3b0 tx=0x7fe5bc009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 shutdown_connections 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe5b406ec00 0x7fe5b40710b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d0066ce0 0x7fe5d010fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 --2- 172.21.15.44:0/3792756458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe5d00ffd60 0x7fe5d010ffb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 >> 172.21.15.44:0/3792756458 conn(0x7fe5d00f7290 msgr2=0x7fe5d00fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 shutdown_connections 2024-04-16T11:52:26.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.689+0000 7fe5d5000700 1 -- 172.21.15.44:0/3792756458 wait complete. 2024-04-16T11:52:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.847+0000 7fa0a5000700 1 -- 172.21.15.44:0/1365550034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fcef0 msgr2=0x7fa0a00fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.847+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1365550034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fcef0 0x7fa0a00fd320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa090004940 tx=0x7fa090009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.847+0000 7fa0a5000700 1 -- 172.21.15.44:0/1365550034 shutdown_connections 2024-04-16T11:52:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.847+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1365550034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.847+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1365550034 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fcef0 0x7fa0a00fd320 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.847+0000 7fa0a5000700 1 -- 172.21.15.44:0/1365550034 >> 172.21.15.44:0/1365550034 conn(0x7fa0a00f8530 msgr2=0x7fa0a00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.848+0000 7fa0a5000700 1 -- 172.21.15.44:0/1365550034 shutdown_connections 2024-04-16T11:52:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.848+0000 7fa0a5000700 1 -- 172.21.15.44:0/1365550034 wait complete. 2024-04-16T11:52:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.849+0000 7fa0a5000700 1 Processor -- start 2024-04-16T11:52:26.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.849+0000 7fa0a5000700 1 -- start start 2024-04-16T11:52:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.849+0000 7fa0a5000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0a00fcef0 0x7fa0a0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.849+0000 7fa0a5000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a0070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.849+0000 7fa0a5000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00708c0 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.849+0000 7fa0a5000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a0070a30 con 0x7fa0a00fcef0 2024-04-16T11:52:26.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09d400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a0070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38538/0 (socket says 172.21.15.44:38538) 2024-04-16T11:52:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09d400700 1 -- 172.21.15.44:0/1890594463 learned_addr learned my addr 172.21.15.44:0/1890594463 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09de00700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0a00fcef0 0x7fa0a0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09d400700 1 -- 172.21.15.44:0/1890594463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0a00fcef0 msgr2=0x7fa0a0071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09d400700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0a00fcef0 0x7fa0a0071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:26.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.850+0000 7fa09d400700 1 -- 172.21.15.44:0/1890594463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa090005040 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.851+0000 7fa09d400700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a0070310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa08c002a50 tx=0x7fa08c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:26.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.851+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08c007cc0 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.851+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa08c00f040 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.851+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08c009640 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.851+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a0070d10 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.851+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a0115ab0 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.853+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a005f5f0 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.854+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa08c0097a0 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.855+0000 7fa096000700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa08406ec10 0x7fa0840710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:26.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.855+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fa08c08e810 con 0x7fa0a00fe0e0 2024-04-16T11:52:26.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.856+0000 7fa09de00700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa08406ec10 0x7fa0840710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:26.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.856+0000 7fa09de00700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa08406ec10 0x7fa0840710c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa090000c00 tx=0x7fa090009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:26.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:26.861+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa08c0573e0 con 0x7fa0a00fe0e0 2024-04-16T11:52:27.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.104+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a0102440 con 0x7fa08406ec10 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.134+0000 7fa096000700 1 -- 172.21.15.44:0/1890594463 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fa0a0102440 con 0x7fa08406ec10 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (9m) 0s ago 19m 13.7M - 0.25.0 c8568f914cd2 778bac6732c2 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (6m) 0s ago 20m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (6m) 29s ago 18m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (9m) 0s ago 19m 73.6M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:52:27.132 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (114s) 0s ago 12m 11.7M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (2m) 0s ago 12m 17.9M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (59s) 29s ago 12m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 e08aedf9cf6c 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (85s) 29s ago 12m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (9m) 0s ago 21m 512M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (8m) 29s ago 18m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (7m) 0s ago 22m 54.7M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (7m) 29s ago 18m 39.8M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (40s) 0s ago 20m 7272k - 1.5.0 0da6a335fe13 d6c7c8a1653a 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (33s) 29s ago 18m 3007k - 1.5.0 0da6a335fe13 e9596e00b617 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (5m) 0s ago 17m 55.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (5m) 0s ago 17m 45.9M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (4m) 0s ago 16m 45.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (4m) 29s ago 15m 45.2M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (3m) 29s ago 15m 42.6M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (2m) 29s ago 14m 49.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:52:27.133 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (8s) 0s ago 19m 41.2M - 2.43.0 a07b618ecd1d b3776ca300db 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa08406ec10 msgr2=0x7fa0840710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa08406ec10 0x7fa0840710c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa090000c00 tx=0x7fa090009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 msgr2=0x7fa0a0070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a0070310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa08c002a50 tx=0x7fa08c0046b0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 shutdown_connections 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa08406ec10 0x7fa0840710c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0a00fcef0 0x7fa0a0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 --2- 172.21.15.44:0/1890594463 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa0a00fe0e0 0x7fa0a0070310 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 >> 172.21.15.44:0/1890594463 conn(0x7fa0a00f8530 msgr2=0x7fa0a00f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 shutdown_connections 2024-04-16T11:52:27.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.137+0000 7fa0a5000700 1 -- 172.21.15.44:0/1890594463 wait complete. 2024-04-16T11:52:27.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.291+0000 7f1f19200700 1 -- 172.21.15.44:0/3986796091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140662f0 msgr2=0x7f1f140ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.291+0000 7f1f19200700 1 --2- 172.21.15.44:0/3986796091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140662f0 0x7f1f140ff800 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1f04004940 tx=0x7f1f04009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.292+0000 7f1f19200700 1 -- 172.21.15.44:0/3986796091 shutdown_connections 2024-04-16T11:52:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.292+0000 7f1f19200700 1 --2- 172.21.15.44:0/3986796091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140ffd40 0x7f1f14102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.292+0000 7f1f19200700 1 --2- 172.21.15.44:0/3986796091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140662f0 0x7f1f140ff800 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.292+0000 7f1f19200700 1 -- 172.21.15.44:0/3986796091 >> 172.21.15.44:0/3986796091 conn(0x7f1f140f7270 msgr2=0x7f1f140f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.292+0000 7f1f19200700 1 -- 172.21.15.44:0/3986796091 shutdown_connections 2024-04-16T11:52:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.292+0000 7f1f19200700 1 -- 172.21.15.44:0/3986796091 wait complete. 2024-04-16T11:52:27.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f19200700 1 Processor -- start 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f19200700 1 -- start start 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f19200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140662f0 0x7f1f1410fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f19200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 0x7f1f1410fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f19200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f141105f0 con 0x7f1f140ffd40 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f19200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14110e40 con 0x7f1f140662f0 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.293+0000 7f1f12a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 0x7f1f1410fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f12a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 0x7f1f1410fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38558/0 (socket says 172.21.15.44:38558) 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f12a00700 1 -- 172.21.15.44:0/665506875 learned_addr learned my addr 172.21.15.44:0/665506875 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f13400700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140662f0 0x7f1f1410fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f12a00700 1 -- 172.21.15.44:0/665506875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140662f0 msgr2=0x7f1f1410fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f12a00700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140662f0 0x7f1f1410fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f12a00700 1 -- 172.21.15.44:0/665506875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04005040 con 0x7f1f140ffd40 2024-04-16T11:52:27.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f13400700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140662f0 0x7f1f1410fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:52:27.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.294+0000 7f1f12a00700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 0x7f1f1410fff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1f00002a50 tx=0x7f1f000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:27.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.295+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f00007cc0 con 0x7f1f140ffd40 2024-04-16T11:52:27.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.295+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f0000f040 con 0x7f1f140ffd40 2024-04-16T11:52:27.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.295+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f14111120 con 0x7f1f140ffd40 2024-04-16T11:52:27.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.295+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f00009640 con 0x7f1f140ffd40 2024-04-16T11:52:27.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.295+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f14111660 con 0x7f1f140ffd40 2024-04-16T11:52:27.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.296+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1405f5f0 con 0x7f1f140ffd40 2024-04-16T11:52:27.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.297+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1f000097a0 con 0x7f1f140ffd40 2024-04-16T11:52:27.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.299+0000 7f1f0b400700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1ef806ec10 0x7f1ef80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:27.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.299+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f1f0008e980 con 0x7f1f140ffd40 2024-04-16T11:52:27.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.299+0000 7f1f13400700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1ef806ec10 0x7f1ef80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:27.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.300+0000 7f1f13400700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1ef806ec10 0x7f1ef80710c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b3b0 tx=0x7f1f04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:27.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.303+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1f0005af60 con 0x7f1f140ffd40 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.632+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1f14111980 con 0x7f1f140ffd40 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.633+0000 7f1f0b400700 1 -- 172.21.15.44:0/665506875 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f1f0005aaf0 con 0x7f1f140ffd40 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:52:27.631 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:52:27.632 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:52:27.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1ef806ec10 msgr2=0x7f1ef80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1ef806ec10 0x7f1ef80710c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b3b0 tx=0x7f1f04009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 msgr2=0x7f1f1410fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 0x7f1f1410fff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1f00002a50 tx=0x7f1f000046b0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 shutdown_connections 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f1ef806ec10 0x7f1ef80710c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f140662f0 0x7f1f1410fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 --2- 172.21.15.44:0/665506875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f140ffd40 0x7f1f1410fff0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 >> 172.21.15.44:0/665506875 conn(0x7f1f140f7270 msgr2=0x7f1f140fb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 shutdown_connections 2024-04-16T11:52:27.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.636+0000 7f1f19200700 1 -- 172.21.15.44:0/665506875 wait complete. 2024-04-16T11:52:27.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.795+0000 7fafdba00700 1 -- 172.21.15.44:0/769934293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40662f0 msgr2=0x7fafd40ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.795+0000 7fafdba00700 1 --2- 172.21.15.44:0/769934293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40662f0 0x7fafd40ffa30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fafc8004940 tx=0x7fafc8009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:27.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.795+0000 7fafdba00700 1 -- 172.21.15.44:0/769934293 shutdown_connections 2024-04-16T11:52:27.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.795+0000 7fafdba00700 1 --2- 172.21.15.44:0/769934293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd40fff70 0x7fafd4102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.795+0000 7fafdba00700 1 --2- 172.21.15.44:0/769934293 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40662f0 0x7fafd40ffa30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.795+0000 7fafdba00700 1 -- 172.21.15.44:0/769934293 >> 172.21.15.44:0/769934293 conn(0x7fafd40f74c0 msgr2=0x7fafd40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:27.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.796+0000 7fafdba00700 1 -- 172.21.15.44:0/769934293 shutdown_connections 2024-04-16T11:52:27.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.796+0000 7fafdba00700 1 -- 172.21.15.44:0/769934293 wait complete. 2024-04-16T11:52:27.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.796+0000 7fafdba00700 1 Processor -- start 2024-04-16T11:52:27.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.797+0000 7fafdba00700 1 -- start start 2024-04-16T11:52:27.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.797+0000 7fafdba00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd40662f0 0x7fafd410fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:27.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.797+0000 7fafdba00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 0x7fafd4110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:27.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.797+0000 7fafdba00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd4110790 con 0x7fafd40fff70 2024-04-16T11:52:27.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.797+0000 7fafdba00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd4110fe0 con 0x7fafd40662f0 2024-04-16T11:52:27.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.797+0000 7fafd3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 0x7fafd4110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 0x7fafd4110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38576/0 (socket says 172.21.15.44:38576) 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd3e00700 1 -- 172.21.15.44:0/1617578119 learned_addr learned my addr 172.21.15.44:0/1617578119 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd9000700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd40662f0 0x7fafd410fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd3e00700 1 -- 172.21.15.44:0/1617578119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd40662f0 msgr2=0x7fafd410fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd3e00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd40662f0 0x7fafd410fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd3e00700 1 -- 172.21.15.44:0/1617578119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc8005040 con 0x7fafd40fff70 2024-04-16T11:52:27.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.798+0000 7fafd3e00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 0x7fafd4110190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c8c0 tx=0x7fafc00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:27.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.799+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafc000a980 con 0x7fafd40fff70 2024-04-16T11:52:27.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.799+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafc000ce60 con 0x7fafd40fff70 2024-04-16T11:52:27.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.799+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd41112c0 con 0x7fafd40fff70 2024-04-16T11:52:27.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.799+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafc00115e0 con 0x7fafd40fff70 2024-04-16T11:52:27.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.799+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd4111800 con 0x7fafd40fff70 2024-04-16T11:52:27.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.800+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd40f8ff0 con 0x7fafd40fff70 2024-04-16T11:52:27.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.801+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fafc0012400 con 0x7fafd40fff70 2024-04-16T11:52:27.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.802+0000 7fafd0c00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fafbc06ec10 0x7fafbc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:27.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.803+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fafc008eb90 con 0x7fafd40fff70 2024-04-16T11:52:27.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.803+0000 7fafd9000700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fafbc06ec10 0x7fafbc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:27.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.804+0000 7fafd9000700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fafbc06ec10 0x7fafbc0710c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009fd0 tx=0x7fafc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:27.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:27.807+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fafc005b270 con 0x7fafd40fff70 2024-04-16T11:52:27.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:27 smithi044 ceph-mon[93020]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:27.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:27 smithi044 ceph-mon[93020]: pgmap v279: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:27.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:27 smithi044 ceph-mon[93020]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:27.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:27 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:27.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:27 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:27.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:27 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/665506875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:28.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:27 smithi045 ceph-mon[81423]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:28.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:27 smithi045 ceph-mon[81423]: pgmap v279: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:28.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:27 smithi045 ceph-mon[81423]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:28.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:27 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:28.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:27 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:28.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:27 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/665506875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.125+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fafd405f5f0 con 0x7fafd40fff70 2024-04-16T11:52:28.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.126+0000 7fafd0c00700 1 -- 172.21.15.44:0/1617578119 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 37 v37) v1 ==== 76+0+1932 (secure 0 0 0) 0x7fafc00126b0 con 0x7fafd40fff70 2024-04-16T11:52:28.123 INFO:teuthology.orchestra.run.smithi044.stdout:e37 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 36 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:51:22.505053+0000 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:52:28.124 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 91 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=34432} 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:52:28.125 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 34432 members: 34432 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:34432} state up:active seq 24 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:44319} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:52:28.126 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:44341} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:52:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fafbc06ec10 msgr2=0x7fafbc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fafbc06ec10 0x7fafbc0710c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009fd0 tx=0x7fafc8009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 msgr2=0x7fafd4110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 0x7fafd4110190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c8c0 tx=0x7fafc00044c0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 shutdown_connections 2024-04-16T11:52:28.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fafbc06ec10 0x7fafbc0710c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafd40662f0 0x7fafd410fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 --2- 172.21.15.44:0/1617578119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fafd40fff70 0x7fafd4110190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.129+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 >> 172.21.15.44:0/1617578119 conn(0x7fafd40f74c0 msgr2=0x7fafd40fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.130+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 shutdown_connections 2024-04-16T11:52:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.130+0000 7fafdba00700 1 -- 172.21.15.44:0/1617578119 wait complete. 2024-04-16T11:52:28.128 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 -- 172.21.15.44:0/3849065580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 msgr2=0x7f9a6c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 --2- 172.21.15.44:0/3849065580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c0fe070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9a58004940 tx=0x7f9a58009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 -- 172.21.15.44:0/3849065580 shutdown_connections 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 --2- 172.21.15.44:0/3849065580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c0fe070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 --2- 172.21.15.44:0/3849065580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 0x7f9a6c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 -- 172.21.15.44:0/3849065580 >> 172.21.15.44:0/3849065580 conn(0x7f9a6c0f8540 msgr2=0x7f9a6c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 -- 172.21.15.44:0/3849065580 shutdown_connections 2024-04-16T11:52:28.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.281+0000 7f9a72600700 1 -- 172.21.15.44:0/3849065580 wait complete. 2024-04-16T11:52:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.282+0000 7f9a72600700 1 Processor -- start 2024-04-16T11:52:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.282+0000 7f9a72600700 1 -- start start 2024-04-16T11:52:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.282+0000 7f9a72600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 0x7f9a6c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.282+0000 7f9a72600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.282+0000 7f9a72600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c111740 con 0x7f9a6c0fd710 2024-04-16T11:52:28.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.282+0000 7f9a72600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c1118b0 con 0x7f9a6c0fcdc0 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38596/0 (socket says 172.21.15.44:38596) 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6aa00700 1 -- 172.21.15.44:0/2559973400 learned_addr learned my addr 172.21.15.44:0/2559973400 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6b400700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 0x7f9a6c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6aa00700 1 -- 172.21.15.44:0/2559973400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 msgr2=0x7f9a6c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6aa00700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 0x7f9a6c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6aa00700 1 -- 172.21.15.44:0/2559973400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58005040 con 0x7f9a6c0fd710 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.283+0000 7f9a6b400700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 0x7f9a6c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:52:28.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.284+0000 7f9a6aa00700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c070310 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b970 tx=0x7f9a58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.284+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5800e040 con 0x7f9a6c0fd710 2024-04-16T11:52:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.284+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a58004290 con 0x7f9a6c0fd710 2024-04-16T11:52:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.284+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c070910 con 0x7f9a6c0fd710 2024-04-16T11:52:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.284+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5801c630 con 0x7f9a6c0fd710 2024-04-16T11:52:28.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.284+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c070e80 con 0x7f9a6c0fd710 2024-04-16T11:52:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.285+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c1060f0 con 0x7f9a6c0fd710 2024-04-16T11:52:28.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.286+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9a5800bbf0 con 0x7f9a6c0fd710 2024-04-16T11:52:28.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.287+0000 7f9a63400700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9a5006ec10 0x7f9a500710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:28.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.288+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f9a5808fe20 con 0x7f9a6c0fd710 2024-04-16T11:52:28.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.288+0000 7f9a6b400700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9a5006ec10 0x7f9a500710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:28.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.289+0000 7f9a6b400700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9a5006ec10 0x7f9a500710c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c071bd0 tx=0x7f9a5c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:28.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.293+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9a580589d0 con 0x7f9a6c0fd710 2024-04-16T11:52:28.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.545+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a6c102490 con 0x7f9a5006ec10 2024-04-16T11:52:28.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.547+0000 7f9a63400700 1 -- 172.21.15.44:0/2559973400 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7f9a6c102490 con 0x7f9a5006ec10 2024-04-16T11:52:28.545 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:52:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:52:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:52:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:52:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:52:28.545 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "mds" 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:52:28.546 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:52:28.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.549+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9a5006ec10 msgr2=0x7f9a500710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.549+0000 7f9a72600700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9a5006ec10 0x7f9a500710c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c071bd0 tx=0x7f9a5c009040 comp rx=0 tx=0).stop 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.549+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 msgr2=0x7f9a6c070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.549+0000 7f9a72600700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c070310 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b970 tx=0x7f9a58009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 shutdown_connections 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9a5006ec10 0x7f9a500710c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a6c0fcdc0 0x7f9a6c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 --2- 172.21.15.44:0/2559973400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9a6c0fd710 0x7f9a6c070310 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 >> 172.21.15.44:0/2559973400 conn(0x7f9a6c0f8540 msgr2=0x7f9a6c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:28.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 shutdown_connections 2024-04-16T11:52:28.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.550+0000 7f9a72600700 1 -- 172.21.15.44:0/2559973400 wait complete. 2024-04-16T11:52:28.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 -- 172.21.15.44:0/239526972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fbb10 msgr2=0x7fdb2c0fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 --2- 172.21.15.44:0/239526972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fbb10 0x7fdb2c0fbf20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c004940 tx=0x7fdb1c009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 -- 172.21.15.44:0/239526972 shutdown_connections 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 --2- 172.21.15.44:0/239526972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 --2- 172.21.15.44:0/239526972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fbb10 0x7fdb2c0fbf20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 -- 172.21.15.44:0/239526972 >> 172.21.15.44:0/239526972 conn(0x7fdb2c0f7270 msgr2=0x7fdb2c0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 -- 172.21.15.44:0/239526972 shutdown_connections 2024-04-16T11:52:28.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.709+0000 7fdb2be00700 1 -- 172.21.15.44:0/239526972 wait complete. 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.710+0000 7fdb2be00700 1 Processor -- start 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.710+0000 7fdb2be00700 1 -- start start 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb2be00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb2c0fbb10 0x7fdb2c0ffb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb2be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb2be00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0fe6d0 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb2be00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0fe810 con 0x7fdb2c0fbb10 2024-04-16T11:52:28.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb23e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb23e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fe100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38608/0 (socket says 172.21.15.44:38608) 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb23e00700 1 -- 172.21.15.44:0/684679578 learned_addr learned my addr 172.21.15.44:0/684679578 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.711+0000 7fdb2aa00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb2c0fbb10 0x7fdb2c0ffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb23e00700 1 -- 172.21.15.44:0/684679578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb2c0fbb10 msgr2=0x7fdb2c0ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb23e00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb2c0fbb10 0x7fdb2c0ffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb23e00700 1 -- 172.21.15.44:0/684679578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c005040 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb23e00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fe100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400b790 tx=0x7fdb14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb1400f770 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb1400fda0 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c0feaf0 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.712+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb1400d310 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.713+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c115960 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.714+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c101950 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.715+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb14010420 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.716+0000 7fdb22a00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdb1006ec10 0x7fdb100710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.716+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fdb1408eaf0 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.716+0000 7fdb2aa00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdb1006ec10 0x7fdb100710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.717+0000 7fdb2aa00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdb1006ec10 0x7fdb100710c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00b3b0 tx=0x7fdb1c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:28.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:28.721+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdb140576a0 con 0x7fdb2c0fcdc0 2024-04-16T11:52:28.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:28 smithi044 ceph-mon[93020]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:28.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:28 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1617578119' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:52:28.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:28 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:28.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:28 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:29.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.063+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb2c05f5f0 con 0x7fdb2c0fcdc0 2024-04-16T11:52:29.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.064+0000 7fdb22a00700 1 -- 172.21.15.44:0/684679578 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fdb1405ad60 con 0x7fdb2c0fcdc0 2024-04-16T11:52:29.062 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdb1006ec10 msgr2=0x7fdb100710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdb1006ec10 0x7fdb100710c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00b3b0 tx=0x7fdb1c009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 msgr2=0x7fdb2c0fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fe100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400b790 tx=0x7fdb14009250 comp rx=0 tx=0).stop 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 shutdown_connections 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fdb1006ec10 0x7fdb100710c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb2c0fbb10 0x7fdb2c0ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 --2- 172.21.15.44:0/684679578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2c0fcdc0 0x7fdb2c0fe100 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 >> 172.21.15.44:0/684679578 conn(0x7fdb2c0f7270 msgr2=0x7fdb2c065b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 shutdown_connections 2024-04-16T11:52:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:29.067+0000 7fdb2be00700 1 -- 172.21.15.44:0/684679578 wait complete. 2024-04-16T11:52:29.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:28 smithi045 ceph-mon[81423]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:29.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:28 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1617578119' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:52:29.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:28 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:29.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:28 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:29.967 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:29 smithi044 ceph-mon[93020]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:29.968 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:29 smithi044 ceph-mon[93020]: pgmap v280: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:29.968 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:29 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/684679578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:52:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:29 smithi045 ceph-mon[81423]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:52:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:29 smithi045 ceph-mon[81423]: pgmap v280: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:29 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/684679578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:52:31.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:31 smithi044 ceph-mon[93020]: pgmap v281: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:32.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:31 smithi045 ceph-mon[81423]: pgmap v281: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:32.746 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 538, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133163 2024-04-16T11:52:32.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 538, success, outsize: 120 2024-04-16T11:52:32.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 540, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133261 2024-04-16T11:52:32.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 540, success, outsize: 120 2024-04-16T11:52:32.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 542, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133362 2024-04-16T11:52:32.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 542, success, outsize: 120 2024-04-16T11:52:32.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 544, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133469 2024-04-16T11:52:32.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 544, success, outsize: 120 2024-04-16T11:52:32.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 546, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133573 2024-04-16T11:52:32.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 546, success, outsize: 120 2024-04-16T11:52:32.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 548, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133671 2024-04-16T11:52:32.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 548, success, outsize: 120 2024-04-16T11:52:32.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 550, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133772 2024-04-16T11:52:32.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 550, success, outsize: 120 2024-04-16T11:52:32.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 552, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133879 2024-04-16T11:52:32.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 552, success, outsize: 120 2024-04-16T11:52:32.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 554, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 133983 2024-04-16T11:52:32.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 554, success, outsize: 120 2024-04-16T11:52:33.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 556, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134081 2024-04-16T11:52:33.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 556, success, outsize: 120 2024-04-16T11:52:33.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 558, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134182 2024-04-16T11:52:33.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 558, success, outsize: 120 2024-04-16T11:52:33.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 560, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134289 2024-04-16T11:52:33.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 560, success, outsize: 120 2024-04-16T11:52:33.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 562, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134393 2024-04-16T11:52:33.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 562, success, outsize: 120 2024-04-16T11:52:33.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 564, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134491 2024-04-16T11:52:33.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 564, success, outsize: 120 2024-04-16T11:52:33.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 566, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134592 2024-04-16T11:52:33.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 566, success, outsize: 120 2024-04-16T11:52:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 568, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134699 2024-04-16T11:52:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 568, success, outsize: 120 2024-04-16T11:52:33.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 570, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134803 2024-04-16T11:52:33.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 570, success, outsize: 120 2024-04-16T11:52:33.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 572, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134901 2024-04-16T11:52:33.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 572, success, outsize: 120 2024-04-16T11:52:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 574, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135002 2024-04-16T11:52:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 574, success, outsize: 120 2024-04-16T11:52:33.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 576, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135109 2024-04-16T11:52:33.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 576, success, outsize: 120 2024-04-16T11:52:33.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 578, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135213 2024-04-16T11:52:33.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 578, success, outsize: 120 2024-04-16T11:52:33.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 580, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135311 2024-04-16T11:52:33.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 580, success, outsize: 120 2024-04-16T11:52:33.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 582, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135412 2024-04-16T11:52:33.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 582, success, outsize: 120 2024-04-16T11:52:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 584, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135519 2024-04-16T11:52:33.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 584, success, outsize: 120 2024-04-16T11:52:33.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 586, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135623 2024-04-16T11:52:33.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 586, success, outsize: 120 2024-04-16T11:52:33.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 588, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135721 2024-04-16T11:52:33.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 588, success, outsize: 120 2024-04-16T11:52:33.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 590, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135822 2024-04-16T11:52:33.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 590, success, outsize: 120 2024-04-16T11:52:33.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 592, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135929 2024-04-16T11:52:33.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 592, success, outsize: 120 2024-04-16T11:52:33.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:33 smithi044 ceph-mon[93020]: pgmap v282: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:34.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:33 smithi045 ceph-mon[81423]: pgmap v282: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:35.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: pgmap v283: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:52:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:52:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: pgmap v283: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:35.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:52:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:37 smithi045 ceph-mon[81423]: pgmap v284: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:38.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:37 smithi044 ceph-mon[93020]: pgmap v284: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:38.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:38 smithi044 ceph-mon[93020]: Upgrade: Updating alertmanager.smithi044 2024-04-16T11:52:38.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:38 smithi044 ceph-mon[93020]: Deploying daemon alertmanager.smithi044 on smithi044 2024-04-16T11:52:39.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:38 smithi045 ceph-mon[81423]: Upgrade: Updating alertmanager.smithi044 2024-04-16T11:52:39.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:38 smithi045 ceph-mon[81423]: Deploying daemon alertmanager.smithi044 on smithi044 2024-04-16T11:52:40.029 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:39 smithi044 ceph-mon[93020]: pgmap v285: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:39 smithi045 ceph-mon[81423]: pgmap v285: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:40.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:41.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:42.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:41 smithi045 ceph-mon[81423]: pgmap v286: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:41 smithi044 ceph-mon[93020]: pgmap v286: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:43.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:43 smithi044 ceph-mon[93020]: pgmap v287: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:44.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:43 smithi045 ceph-mon[81423]: pgmap v287: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:45.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:45 smithi044 ceph-mon[93020]: pgmap v288: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:46.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:45 smithi045 ceph-mon[81423]: pgmap v288: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:47.296 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:47.297 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:47.297 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:47.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:46 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:47.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:46 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:47.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:46 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:52:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:47 smithi044 ceph-mon[93020]: pgmap v289: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:47 smithi045 ceph-mon[81423]: pgmap v289: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:50.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:49 smithi045 ceph-mon[81423]: pgmap v290: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:50.402 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:49 smithi044 ceph-mon[93020]: pgmap v290: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:51.316 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:51 smithi044 ceph-mon[93020]: pgmap v291: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:51.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:51 smithi045 ceph-mon[81423]: pgmap v291: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:53 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:53.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:53 smithi044 ceph-mon[93020]: pgmap v292: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:53.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:53.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:53 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:53.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:53 smithi045 ceph-mon[81423]: pgmap v292: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:55.813 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:55.813 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:55.813 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:55 smithi044 ceph-mon[93020]: pgmap v293: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:55.813 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:52:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:55 smithi045 ceph-mon[81423]: pgmap v293: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:55.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:52:58.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:57 smithi045 ceph-mon[81423]: pgmap v294: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:58.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:57 smithi044 ceph-mon[93020]: pgmap v294: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:58.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 594, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137569 2024-04-16T11:52:58.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 594, success, outsize: 120 2024-04-16T11:52:58.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 596, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137662 2024-04-16T11:52:58.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 596, success, outsize: 120 2024-04-16T11:52:58.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 598, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137758 2024-04-16T11:52:58.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 598, success, outsize: 120 2024-04-16T11:52:58.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 600, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137860 2024-04-16T11:52:58.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 600, success, outsize: 120 2024-04-16T11:52:58.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 602, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 137959 2024-04-16T11:52:58.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 602, success, outsize: 120 2024-04-16T11:52:58.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 604, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138052 2024-04-16T11:52:58.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 604, success, outsize: 120 2024-04-16T11:52:58.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 606, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138148 2024-04-16T11:52:58.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 606, success, outsize: 120 2024-04-16T11:52:58.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 608, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138250 2024-04-16T11:52:58.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 608, success, outsize: 120 2024-04-16T11:52:58.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 610, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138349 2024-04-16T11:52:58.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 610, success, outsize: 120 2024-04-16T11:52:58.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 612, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138442 2024-04-16T11:52:58.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 612, success, outsize: 120 2024-04-16T11:52:59.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 614, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138538 2024-04-16T11:52:59.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 614, success, outsize: 120 2024-04-16T11:52:59.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 616, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138640 2024-04-16T11:52:59.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 616, success, outsize: 120 2024-04-16T11:52:59.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 618, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138739 2024-04-16T11:52:59.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 618, success, outsize: 120 2024-04-16T11:52:59.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 620, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138835 2024-04-16T11:52:59.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 620, success, outsize: 120 2024-04-16T11:52:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 622, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138931 2024-04-16T11:52:59.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 622, success, outsize: 120 2024-04-16T11:52:59.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 624, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139033 2024-04-16T11:52:59.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 624, success, outsize: 120 2024-04-16T11:52:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 626, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139132 2024-04-16T11:52:59.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 626, success, outsize: 120 2024-04-16T11:52:59.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 628, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139225 2024-04-16T11:52:59.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 628, success, outsize: 120 2024-04-16T11:52:59.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 630, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139321 2024-04-16T11:52:59.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 630, success, outsize: 120 2024-04-16T11:52:59.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 632, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139424 2024-04-16T11:52:59.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 632, success, outsize: 120 2024-04-16T11:52:59.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.233+0000 7ff782000700 1 -- 172.21.15.44:0/2623384147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c103420 msgr2=0x7ff77c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.233+0000 7ff782000700 1 --2- 172.21.15.44:0/2623384147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c103420 0x7ff77c103830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff76c004940 tx=0x7ff76c009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 -- 172.21.15.44:0/2623384147 shutdown_connections 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 --2- 172.21.15.44:0/2623384147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c1046d0 0x7ff77c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 --2- 172.21.15.44:0/2623384147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c103420 0x7ff77c103830 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 -- 172.21.15.44:0/2623384147 >> 172.21.15.44:0/2623384147 conn(0x7ff77c072640 msgr2=0x7ff77c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 -- 172.21.15.44:0/2623384147 shutdown_connections 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 -- 172.21.15.44:0/2623384147 wait complete. 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.234+0000 7ff782000700 1 Processor -- start 2024-04-16T11:52:59.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.235+0000 7ff782000700 1 -- start start 2024-04-16T11:52:59.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.235+0000 7ff782000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c103420 0x7ff77c113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:59.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.235+0000 7ff782000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 0x7ff77c1144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:59.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.235+0000 7ff782000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c114ac0 con 0x7ff77c1046d0 2024-04-16T11:52:59.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.235+0000 7ff782000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c114c30 con 0x7ff77c103420 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.235+0000 7ff77aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 0x7ff77c1144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 0x7ff77c1144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39452/0 (socket says 172.21.15.44:39452) 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77aa00700 1 -- 172.21.15.44:0/743115581 learned_addr learned my addr 172.21.15.44:0/743115581 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77b400700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c103420 0x7ff77c113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77aa00700 1 -- 172.21.15.44:0/743115581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c103420 msgr2=0x7ff77c113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77aa00700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c103420 0x7ff77c113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77aa00700 1 -- 172.21.15.44:0/743115581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c005040 con 0x7ff77c1046d0 2024-04-16T11:52:59.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.236+0000 7ff77aa00700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 0x7ff77c1144c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c810 tx=0x7ff768004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.237+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff768012810 con 0x7ff77c1046d0 2024-04-16T11:52:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.237+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff768012e40 con 0x7ff77c1046d0 2024-04-16T11:52:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.237+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c119690 con 0x7ff77c1046d0 2024-04-16T11:52:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.237+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff76800a370 con 0x7ff77c1046d0 2024-04-16T11:52:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.237+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c119c50 con 0x7ff77c1046d0 2024-04-16T11:52:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.239+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c04c710 con 0x7ff77c1046d0 2024-04-16T11:52:59.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.239+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff768011810 con 0x7ff77c1046d0 2024-04-16T11:52:59.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.240+0000 7ff773400700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff76006ec00 0x7ff7600710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:59.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.241+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7ff76808eed0 con 0x7ff77c1046d0 2024-04-16T11:52:59.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.241+0000 7ff77b400700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff76006ec00 0x7ff7600710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:59.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.242+0000 7ff77b400700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff76006ec00 0x7ff7600710b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff76c004f90 tx=0x7ff76c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:59.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 634, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139542 2024-04-16T11:52:59.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 634, success, outsize: 120 2024-04-16T11:52:59.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.246+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff76805b530 con 0x7ff77c1046d0 2024-04-16T11:52:59.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 636, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139635 2024-04-16T11:52:59.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 636, success, outsize: 120 2024-04-16T11:52:59.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 638, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139731 2024-04-16T11:52:59.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 638, success, outsize: 120 2024-04-16T11:52:59.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 640, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139833 2024-04-16T11:52:59.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 640, success, outsize: 120 2024-04-16T11:52:59.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 642, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139932 2024-04-16T11:52:59.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 642, success, outsize: 120 2024-04-16T11:52:59.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 644, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140025 2024-04-16T11:52:59.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 644, success, outsize: 120 2024-04-16T11:52:59.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 646, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140121 2024-04-16T11:52:59.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 646, success, outsize: 120 2024-04-16T11:52:59.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 648, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140223 2024-04-16T11:52:59.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 648, success, outsize: 120 2024-04-16T11:52:59.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.541+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff77c108a60 con 0x7ff76006ec00 2024-04-16T11:52:59.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.544+0000 7ff773400700 1 -- 172.21.15.44:0/743115581 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+440 (secure 0 0 0) 0x7ff77c108a60 con 0x7ff76006ec00 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.546+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff76006ec00 msgr2=0x7ff7600710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.546+0000 7ff782000700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff76006ec00 0x7ff7600710b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff76c004f90 tx=0x7ff76c009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.546+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 msgr2=0x7ff77c1144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.546+0000 7ff782000700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 0x7ff77c1144c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c810 tx=0x7ff768004470 comp rx=0 tx=0).stop 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 shutdown_connections 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff76006ec00 0x7ff7600710b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff77c103420 0x7ff77c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 --2- 172.21.15.44:0/743115581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff77c1046d0 0x7ff77c1144c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 >> 172.21.15.44:0/743115581 conn(0x7ff77c072640 msgr2=0x7ff77c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:59.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 shutdown_connections 2024-04-16T11:52:59.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.547+0000 7ff782000700 1 -- 172.21.15.44:0/743115581 wait complete. 2024-04-16T11:52:59.561 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:52:59.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 -- 172.21.15.44:0/3247845475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 msgr2=0x7ff2200ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 --2- 172.21.15.44:0/3247845475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff2200ff820 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff214004940 tx=0x7ff214009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:59.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 -- 172.21.15.44:0/3247845475 shutdown_connections 2024-04-16T11:52:59.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 --2- 172.21.15.44:0/3247845475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2200ffd60 0x7ff220102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 --2- 172.21.15.44:0/3247845475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff2200ff820 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 -- 172.21.15.44:0/3247845475 >> 172.21.15.44:0/3247845475 conn(0x7ff2200f7290 msgr2=0x7ff2200f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:59.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 -- 172.21.15.44:0/3247845475 shutdown_connections 2024-04-16T11:52:59.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.706+0000 7ff226000700 1 -- 172.21.15.44:0/3247845475 wait complete. 2024-04-16T11:52:59.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.707+0000 7ff226000700 1 Processor -- start 2024-04-16T11:52:59.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.707+0000 7ff226000700 1 -- start start 2024-04-16T11:52:59.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff226000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff22010d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:59.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff226000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2200ffd60 0x7ff22010ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:59.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff226000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22010e3f0 con 0x7ff220066d40 2024-04-16T11:52:59.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff226000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22010ebf0 con 0x7ff2200ffd60 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff224c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff22010d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff224c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff22010d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39456/0 (socket says 172.21.15.44:39456) 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff224c00700 1 -- 172.21.15.44:0/3767108402 learned_addr learned my addr 172.21.15.44:0/3767108402 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff21fe00700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2200ffd60 0x7ff22010ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff224c00700 1 -- 172.21.15.44:0/3767108402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2200ffd60 msgr2=0x7ff22010ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.708+0000 7ff224c00700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2200ffd60 0x7ff22010ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff224c00700 1 -- 172.21.15.44:0/3767108402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214005040 con 0x7ff220066d40 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff224c00700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff22010d8b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b970 tx=0x7ff214009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff21400e040 con 0x7ff220066d40 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff214004290 con 0x7ff220066d40 2024-04-16T11:52:59.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff21401c630 con 0x7ff220066d40 2024-04-16T11:52:59.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22010ee70 con 0x7ff220066d40 2024-04-16T11:52:59.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.709+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22010f3b0 con 0x7ff220066d40 2024-04-16T11:52:59.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.711+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2200faf30 con 0x7ff220066d40 2024-04-16T11:52:59.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.712+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff21400bbf0 con 0x7ff220066d40 2024-04-16T11:52:59.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.713+0000 7ff21cc00700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff20806ec00 0x7ff2080710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:52:59.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.713+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7ff21408fe20 con 0x7ff220066d40 2024-04-16T11:52:59.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.713+0000 7ff21fe00700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff20806ec00 0x7ff2080710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:52:59.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.714+0000 7ff21fe00700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff20806ec00 0x7ff2080710b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff20c007fd0 tx=0x7ff20c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:52:59.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.718+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff21405c500 con 0x7ff220066d40 2024-04-16T11:52:59.804 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:52:59 smithi044 ceph-mon[93020]: pgmap v295: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:52:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.983+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff22010f750 con 0x7ff20806ec00 2024-04-16T11:52:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.986+0000 7ff21cc00700 1 -- 172.21.15.44:0/3767108402 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+440 (secure 0 0 0) 0x7ff22010f750 con 0x7ff20806ec00 2024-04-16T11:52:59.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.989+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff20806ec00 msgr2=0x7ff2080710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.989+0000 7ff226000700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff20806ec00 0x7ff2080710b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff20c007fd0 tx=0x7ff20c009040 comp rx=0 tx=0).stop 2024-04-16T11:52:59.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.989+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 msgr2=0x7ff22010d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:52:59.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.989+0000 7ff226000700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff22010d8b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b970 tx=0x7ff214009e30 comp rx=0 tx=0).stop 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.989+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 shutdown_connections 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.990+0000 7ff226000700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff20806ec00 0x7ff2080710b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.990+0000 7ff226000700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff220066d40 0x7ff22010d8b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.990+0000 7ff226000700 1 --2- 172.21.15.44:0/3767108402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2200ffd60 0x7ff22010ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.990+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 >> 172.21.15.44:0/3767108402 conn(0x7ff2200f7290 msgr2=0x7ff2200f76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.990+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 shutdown_connections 2024-04-16T11:52:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:52:59.990+0000 7ff226000700 1 -- 172.21.15.44:0/3767108402 wait complete. 2024-04-16T11:53:00.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:52:59 smithi045 ceph-mon[81423]: pgmap v295: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:00.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.144+0000 7feef2a00700 1 -- 172.21.15.44:0/2301450754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 msgr2=0x7feeec102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.144+0000 7feef2a00700 1 --2- 172.21.15.44:0/2301450754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec102390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feed4004940 tx=0x7feed4009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:00.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.145+0000 7feef2a00700 1 -- 172.21.15.44:0/2301450754 shutdown_connections 2024-04-16T11:53:00.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.145+0000 7feef2a00700 1 --2- 172.21.15.44:0/2301450754 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec102390 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.145+0000 7feef2a00700 1 --2- 172.21.15.44:0/2301450754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeec0662f0 0x7feeec0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.145+0000 7feef2a00700 1 -- 172.21.15.44:0/2301450754 >> 172.21.15.44:0/2301450754 conn(0x7feeec0f74c0 msgr2=0x7feeec0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:00.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.145+0000 7feef2a00700 1 -- 172.21.15.44:0/2301450754 shutdown_connections 2024-04-16T11:53:00.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.145+0000 7feef2a00700 1 -- 172.21.15.44:0/2301450754 wait complete. 2024-04-16T11:53:00.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.146+0000 7feef2a00700 1 Processor -- start 2024-04-16T11:53:00.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.146+0000 7feef2a00700 1 -- start start 2024-04-16T11:53:00.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.146+0000 7feef2a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeec0662f0 0x7feeec10fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:00.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.146+0000 7feef2a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec110120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:00.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.146+0000 7feef2a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec110720 con 0x7feeec0fff70 2024-04-16T11:53:00.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.146+0000 7feef2a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec110f70 con 0x7feeec0662f0 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feee3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec110120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feee3e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec110120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39468/0 (socket says 172.21.15.44:39468) 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feee3e00700 1 -- 172.21.15.44:0/339500693 learned_addr learned my addr 172.21.15.44:0/339500693 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feeebe00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeec0662f0 0x7feeec10fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feee3e00700 1 -- 172.21.15.44:0/339500693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeec0662f0 msgr2=0x7feeec10fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feee3e00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeec0662f0 0x7feeec10fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.147+0000 7feee3e00700 1 -- 172.21.15.44:0/339500693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4005040 con 0x7feeec0fff70 2024-04-16T11:53:00.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.148+0000 7feee3e00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec110120 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7feed400b0d0 tx=0x7feed4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:00.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.148+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed400e070 con 0x7feeec0fff70 2024-04-16T11:53:00.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.148+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed400b410 con 0x7feeec0fff70 2024-04-16T11:53:00.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.148+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec1111f0 con 0x7feeec0fff70 2024-04-16T11:53:00.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.148+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec111730 con 0x7feeec0fff70 2024-04-16T11:53:00.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.148+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed4027620 con 0x7feeec0fff70 2024-04-16T11:53:00.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.150+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec05f5f0 con 0x7feeec0fff70 2024-04-16T11:53:00.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.151+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feed400ba80 con 0x7feeec0fff70 2024-04-16T11:53:00.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.152+0000 7feee8c00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7feed006ec10 0x7feed00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:00.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.152+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7feed408f7a0 con 0x7feeec0fff70 2024-04-16T11:53:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.152+0000 7feeebe00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7feed006ec10 0x7feed00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.153+0000 7feeebe00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7feed006ec10 0x7feed00710c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feedc004600 tx=0x7feedc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:00.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.157+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7feed405bf50 con 0x7feeec0fff70 2024-04-16T11:53:00.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.435+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feeec111a10 con 0x7feed006ec10 2024-04-16T11:53:00.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.464+0000 7feee8c00700 1 -- 172.21.15.44:0/339500693 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7feeec111a10 con 0x7feed006ec10 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (15s) 8s ago 20m 14.2M - 0.25.0 c8568f914cd2 42392ee6c260 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (6m) 8s ago 20m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (6m) 63s ago 19m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (10m) 8s ago 20m 73.6M - 9.4.7 954c08fa6188 5c40fad8e2f5 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (2m) 8s ago 13m 11.8M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (3m) 8s ago 13m 19.6M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (92s) 63s ago 13m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 e08aedf9cf6c 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (119s) 63s ago 13m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:53:00.462 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (9m) 8s ago 22m 512M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (8m) 63s ago 19m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (8m) 8s ago 22m 55.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (7m) 63s ago 19m 39.8M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (73s) 8s ago 20m 7856k - 1.5.0 0da6a335fe13 d6c7c8a1653a 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (67s) 63s ago 19m 3007k - 1.5.0 0da6a335fe13 e9596e00b617 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (6m) 8s ago 18m 55.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (5m) 8s ago 17m 46.0M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (5m) 8s ago 16m 45.4M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (4m) 63s ago 16m 45.2M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (4m) 63s ago 15m 42.6M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (3m) 63s ago 15m 49.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:53:00.463 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (41s) 8s ago 19m 46.6M - 2.43.0 a07b618ecd1d b3776ca300db 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7feed006ec10 msgr2=0x7feed00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7feed006ec10 0x7feed00710c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feedc004600 tx=0x7feedc00b040 comp rx=0 tx=0).stop 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 msgr2=0x7feeec110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec110120 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7feed400b0d0 tx=0x7feed4009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 shutdown_connections 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7feed006ec10 0x7feed00710c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeec0662f0 0x7feeec10fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 --2- 172.21.15.44:0/339500693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feeec0fff70 0x7feeec110120 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 >> 172.21.15.44:0/339500693 conn(0x7feeec0f74c0 msgr2=0x7feeec0fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.468+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 shutdown_connections 2024-04-16T11:53:00.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.469+0000 7feef2a00700 1 -- 172.21.15.44:0/339500693 wait complete. 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 -- 172.21.15.44:0/1452585306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 msgr2=0x7fd8c81047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1452585306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c81047b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4004940 tx=0x7fd8b4009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 -- 172.21.15.44:0/1452585306 shutdown_connections 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1452585306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c81047b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1452585306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8066290 0x7fd8c8101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 -- 172.21.15.44:0/1452585306 >> 172.21.15.44:0/1452585306 conn(0x7fd8c80f74c0 msgr2=0x7fd8c80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 -- 172.21.15.44:0/1452585306 shutdown_connections 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.628+0000 7fd8cd400700 1 -- 172.21.15.44:0/1452585306 wait complete. 2024-04-16T11:53:00.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.629+0000 7fd8cd400700 1 Processor -- start 2024-04-16T11:53:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.629+0000 7fd8cd400700 1 -- start start 2024-04-16T11:53:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8cd400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8066290 0x7fd8c810fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8cd400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c8110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8cd400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8110770 con 0x7fd8c81023d0 2024-04-16T11:53:00.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8cd400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8110fc0 con 0x7fd8c8066290 2024-04-16T11:53:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8c5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c8110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8c5e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c8110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39476/0 (socket says 172.21.15.44:39476) 2024-04-16T11:53:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8c5e00700 1 -- 172.21.15.44:0/1812996735 learned_addr learned my addr 172.21.15.44:0/1812996735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.630+0000 7fd8c6800700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8066290 0x7fd8c810fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:00.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8c5e00700 1 -- 172.21.15.44:0/1812996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8066290 msgr2=0x7fd8c810fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8c5e00700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8066290 0x7fd8c810fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8c5e00700 1 -- 172.21.15.44:0/1812996735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4005040 con 0x7fd8c81023d0 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8c5e00700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c8110170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400b3b0 tx=0x7fd8b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8b400e040 con 0x7fd8c81023d0 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8b4004290 con 0x7fd8c81023d0 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c8111240 con 0x7fd8c81023d0 2024-04-16T11:53:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8b401c630 con 0x7fd8c81023d0 2024-04-16T11:53:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.631+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c8111780 con 0x7fd8c81023d0 2024-04-16T11:53:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.633+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c805f5f0 con 0x7fd8c81023d0 2024-04-16T11:53:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.634+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd8b400bbf0 con 0x7fd8c81023d0 2024-04-16T11:53:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.635+0000 7fd8bea00700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd8ac06ec10 0x7fd8ac0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.635+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fd8b408fef0 con 0x7fd8c81023d0 2024-04-16T11:53:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.635+0000 7fd8c6800700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd8ac06ec10 0x7fd8ac0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.636+0000 7fd8c6800700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd8ac06ec10 0x7fd8ac0710c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009fd0 tx=0x7fd8b8009f00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.643+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd8b405c5d0 con 0x7fd8c81023d0 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.986+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd8c8111b50 con 0x7fd8c81023d0 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.987+0000 7fd8bea00700 1 -- 172.21.15.44:0/1812996735 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd8b405c160 con 0x7fd8c81023d0 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:53:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:53:00.985 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:00.985 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:53:00.985 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T11:53:00.985 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:53:00.985 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:53:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.990+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd8ac06ec10 msgr2=0x7fd8ac0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.990+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd8ac06ec10 0x7fd8ac0710c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009fd0 tx=0x7fd8b8009f00 comp rx=0 tx=0).stop 2024-04-16T11:53:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.990+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 msgr2=0x7fd8c8110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.990+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c8110170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400b3b0 tx=0x7fd8b4009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 shutdown_connections 2024-04-16T11:53:00.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd8ac06ec10 0x7fd8ac0710c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8c8066290 0x7fd8c810fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 --2- 172.21.15.44:0/1812996735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8c81023d0 0x7fd8c8110170 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:00.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 >> 172.21.15.44:0/1812996735 conn(0x7fd8c80f74c0 msgr2=0x7fd8c80f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:00.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 shutdown_connections 2024-04-16T11:53:00.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:00.991+0000 7fd8cd400700 1 -- 172.21.15.44:0/1812996735 wait complete. 2024-04-16T11:53:01.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:01.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:01.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:01.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:01.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:01.034 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:01.034 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:00 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:01.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:00 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 -- 172.21.15.44:0/1456244900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b41002b0 msgr2=0x7f47b4100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 --2- 172.21.15.44:0/1456244900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b41002b0 0x7f47b4100720 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f47a0004940 tx=0x7f47a0009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 -- 172.21.15.44:0/1456244900 shutdown_connections 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 --2- 172.21.15.44:0/1456244900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b41002b0 0x7f47b4100720 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 --2- 172.21.15.44:0/1456244900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b40ff000 0x7f47b40ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 -- 172.21.15.44:0/1456244900 >> 172.21.15.44:0/1456244900 conn(0x7f47b40fa950 msgr2=0x7f47b40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 -- 172.21.15.44:0/1456244900 shutdown_connections 2024-04-16T11:53:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.152+0000 7f47bb800700 1 -- 172.21.15.44:0/1456244900 wait complete. 2024-04-16T11:53:01.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.153+0000 7f47bb800700 1 Processor -- start 2024-04-16T11:53:01.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.153+0000 7f47bb800700 1 -- start start 2024-04-16T11:53:01.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.153+0000 7f47bb800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 0x7f47b410fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:01.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.153+0000 7f47bb800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b41002b0 0x7f47b4110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:01.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.153+0000 7f47bb800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4110770 con 0x7f47b40ff000 2024-04-16T11:53:01.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.153+0000 7f47bb800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4110fc0 con 0x7f47b41002b0 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 0x7f47b410fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 0x7f47b410fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39494/0 (socket says 172.21.15.44:39494) 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 -- 172.21.15.44:0/2225877024 learned_addr learned my addr 172.21.15.44:0/2225877024 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47abe00700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b41002b0 0x7f47b4110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 -- 172.21.15.44:0/2225877024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b41002b0 msgr2=0x7f47b4110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b41002b0 0x7f47b4110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 -- 172.21.15.44:0/2225877024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a0005040 con 0x7f47b40ff000 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47abe00700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b41002b0 0x7f47b4110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.154+0000 7f47b8e00700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 0x7f47b410fc30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c8c0 tx=0x7f47a40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.155+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a400a980 con 0x7f47b40ff000 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.155+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47a400ce60 con 0x7f47b40ff000 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.155+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b41112a0 con 0x7f47b40ff000 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.155+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a40115e0 con 0x7f47b40ff000 2024-04-16T11:53:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.155+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b41117e0 con 0x7f47b40ff000 2024-04-16T11:53:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.156+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b405f5f0 con 0x7f47b40ff000 2024-04-16T11:53:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.157+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47a4012400 con 0x7f47b40ff000 2024-04-16T11:53:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.158+0000 7f47b1600700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f479806ec10 0x7f47980710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.158+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f47a408eb90 con 0x7f47b40ff000 2024-04-16T11:53:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.159+0000 7f47abe00700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f479806ec10 0x7f47980710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.159+0000 7f47abe00700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f479806ec10 0x7f47980710c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b7a0 tx=0x7f47a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.168+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f47a405b270 con 0x7f47b40ff000 2024-04-16T11:53:01.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.451+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f47b4111ae0 con 0x7f47b40ff000 2024-04-16T11:53:01.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.452+0000 7f47b1600700 1 -- 172.21.15.44:0/2225877024 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 37 v37) v1 ==== 76+0+1932 (secure 0 0 0) 0x7f47a40126b0 con 0x7f47b40ff000 2024-04-16T11:53:01.449 INFO:teuthology.orchestra.run.smithi044.stdout:e37 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 36 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:51:22.505053+0000 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:53:01.450 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 91 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=34432} 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:53:01.451 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 34432 members: 34432 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:34432} state up:active seq 24 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:44319} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:01.452 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:44341} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.455+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f479806ec10 msgr2=0x7f47980710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.455+0000 7f47bb800700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f479806ec10 0x7f47980710c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b7a0 tx=0x7f47a0009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.455+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 msgr2=0x7f47b410fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.455+0000 7f47bb800700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 0x7f47b410fc30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c8c0 tx=0x7f47a40044c0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 shutdown_connections 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f479806ec10 0x7f47980710c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f47b40ff000 0x7f47b410fc30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 --2- 172.21.15.44:0/2225877024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47b41002b0 0x7f47b4110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 >> 172.21.15.44:0/2225877024 conn(0x7f47b40fa950 msgr2=0x7f47b40fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 shutdown_connections 2024-04-16T11:53:01.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.456+0000 7f47bb800700 1 -- 172.21.15.44:0/2225877024 wait complete. 2024-04-16T11:53:01.454 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-16T11:53:01.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.632+0000 7f0101c00700 1 -- 172.21.15.44:0/201696599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc066f00 msgr2=0x7f00fc0ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.632+0000 7f0101c00700 1 --2- 172.21.15.44:0/201696599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc066f00 0x7f00fc0ff9e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f00ec004940 tx=0x7f00ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.633+0000 7f0101c00700 1 -- 172.21.15.44:0/201696599 shutdown_connections 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.633+0000 7f0101c00700 1 --2- 172.21.15.44:0/201696599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00fc0fff20 0x7f00fc102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.633+0000 7f0101c00700 1 --2- 172.21.15.44:0/201696599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc066f00 0x7f00fc0ff9e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.633+0000 7f0101c00700 1 -- 172.21.15.44:0/201696599 >> 172.21.15.44:0/201696599 conn(0x7f00fc0f7450 msgr2=0x7f00fc0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.633+0000 7f0101c00700 1 -- 172.21.15.44:0/201696599 shutdown_connections 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.633+0000 7f0101c00700 1 -- 172.21.15.44:0/201696599 wait complete. 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.634+0000 7f0101c00700 1 Processor -- start 2024-04-16T11:53:01.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.634+0000 7f0101c00700 1 -- start start 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.634+0000 7f0101c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00fc066f00 0x7f00fc10fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.634+0000 7f0101c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 0x7f00fc1100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.634+0000 7f0101c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1106e0 con 0x7f00fc0fff20 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.634+0000 7f0101c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc110850 con 0x7f00fc066f00 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00fa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 0x7f00fc1100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00fa000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 0x7f00fc1100e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47112/0 (socket says 172.21.15.44:47112) 2024-04-16T11:53:01.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00fa000700 1 -- 172.21.15.44:0/1825722378 learned_addr learned my addr 172.21.15.44:0/1825722378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:01.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00faa00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00fc066f00 0x7f00fc10fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:01.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00fa000700 1 -- 172.21.15.44:0/1825722378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00fc066f00 msgr2=0x7f00fc10fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00fa000700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00fc066f00 0x7f00fc10fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.635+0000 7f00fa000700 1 -- 172.21.15.44:0/1825722378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec005040 con 0x7f00fc0fff20 2024-04-16T11:53:01.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.636+0000 7f00fa000700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 0x7f00fc1100e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00e8002a50 tx=0x7f00e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:01.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.636+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00e8007cc0 con 0x7f00fc0fff20 2024-04-16T11:53:01.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.636+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00e800f040 con 0x7f00fc0fff20 2024-04-16T11:53:01.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.636+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc1152b0 con 0x7f00fc0fff20 2024-04-16T11:53:01.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.636+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00e8009640 con 0x7f00fc0fff20 2024-04-16T11:53:01.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.636+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc115870 con 0x7f00fc0fff20 2024-04-16T11:53:01.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.637+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc04c710 con 0x7f00fc0fff20 2024-04-16T11:53:01.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.639+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f00e80097a0 con 0x7f00fc0fff20 2024-04-16T11:53:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.640+0000 7f00f2a00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f00e006ec00 0x7f00e00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.640+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f00e808e980 con 0x7f00fc0fff20 2024-04-16T11:53:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.640+0000 7f00faa00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f00e006ec00 0x7f00e00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.641+0000 7f00faa00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f00e006ec00 0x7f00e00710b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00b970 tx=0x7f00ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:01.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.648+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f00e805af60 con 0x7f00fc0fff20 2024-04-16T11:53:01.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.911+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00fc0fd1e0 con 0x7f00e006ec00 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.913+0000 7f00f2a00700 1 -- 172.21.15.44:0/1825722378 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7f00fc0fd1e0 con 0x7f00e006ec00 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-04-16T11:53:01.910 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "mds" 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading grafana daemons", 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:53:01.911 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f00e006ec00 msgr2=0x7f00e00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f00e006ec00 0x7f00e00710b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00b970 tx=0x7f00ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 msgr2=0x7f00fc1100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 0x7f00fc1100e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00e8002a50 tx=0x7f00e80046b0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 shutdown_connections 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f00e006ec00 0x7f00e00710b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00fc066f00 0x7f00fc10fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 --2- 172.21.15.44:0/1825722378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f00fc0fff20 0x7f00fc1100e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:01.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 >> 172.21.15.44:0/1825722378 conn(0x7f00fc0f7450 msgr2=0x7f00fc0fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:01.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.917+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 shutdown_connections 2024-04-16T11:53:01.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:01.918+0000 7f0101c00700 1 -- 172.21.15.44:0/1825722378 wait complete. 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: pgmap v296: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.062 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1812996735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.063 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:01 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2225877024' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:53:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 -- 172.21.15.44:0/4120709205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 msgr2=0x7f3308104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 --2- 172.21.15.44:0/4120709205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308104b40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f32f4004940 tx=0x7f32f4009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 -- 172.21.15.44:0/4120709205 shutdown_connections 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 --2- 172.21.15.44:0/4120709205 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308104b40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 --2- 172.21.15.44:0/4120709205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3308103420 0x7f3308103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 -- 172.21.15.44:0/4120709205 >> 172.21.15.44:0/4120709205 conn(0x7f3308072640 msgr2=0x7f3308074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.066+0000 7f3310600700 1 -- 172.21.15.44:0/4120709205 shutdown_connections 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.067+0000 7f3310600700 1 -- 172.21.15.44:0/4120709205 wait complete. 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.067+0000 7f3310600700 1 Processor -- start 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f3310600700 1 -- start start 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f3310600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3308103420 0x7f3308113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f3310600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f3310600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3308114a90 con 0x7f33081046d0 2024-04-16T11:53:02.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f3310600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3308114c00 con 0x7f3308103420 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f330d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f330d200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47122/0 (socket says 172.21.15.44:47122) 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.068+0000 7f330d200700 1 -- 172.21.15.44:0/2538292167 learned_addr learned my addr 172.21.15.44:0/2538292167 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.069+0000 7f330dc00700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3308103420 0x7f3308113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.069+0000 7f330d200700 1 -- 172.21.15.44:0/2538292167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3308103420 msgr2=0x7f3308113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.069+0000 7f330d200700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3308103420 0x7f3308113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:02.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.069+0000 7f330d200700 1 -- 172.21.15.44:0/2538292167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4005040 con 0x7f33081046d0 2024-04-16T11:53:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.069+0000 7f330d200700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308114490 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f32f400b3b0 tx=0x7f32f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.070+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f400e040 con 0x7f33081046d0 2024-04-16T11:53:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.070+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32f4004290 con 0x7f33081046d0 2024-04-16T11:53:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.070+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f401c3f0 con 0x7f33081046d0 2024-04-16T11:53:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.070+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3308119600 con 0x7f33081046d0 2024-04-16T11:53:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.070+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3308119bc0 con 0x7f33081046d0 2024-04-16T11:53:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.072+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e80052e0 con 0x7f33081046d0 2024-04-16T11:53:02.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.074+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32f400bbf0 con 0x7f33081046d0 2024-04-16T11:53:02.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.076+0000 7f3305600700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32f006ec00 0x7f32f00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.076+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f32f408fc50 con 0x7f33081046d0 2024-04-16T11:53:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.076+0000 7f330dc00700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32f006ec00 0x7f32f00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:02.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.077+0000 7f330dc00700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32f006ec00 0x7f32f00710b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f32fc004f40 tx=0x7f32fc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:02.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.080+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f32f405c3f0 con 0x7f33081046d0 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: pgmap v296: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1812996735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:02.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:01 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2225877024' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:53:02.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.398+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f32e8005590 con 0x7f33081046d0 2024-04-16T11:53:02.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.399+0000 7f3305600700 1 -- 172.21.15.44:0/2538292167 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f32f400bea0 con 0x7f33081046d0 2024-04-16T11:53:02.395 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:53:02.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32f006ec00 msgr2=0x7f32f00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32f006ec00 0x7f32f00710b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f32fc004f40 tx=0x7f32fc00b040 comp rx=0 tx=0).stop 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 msgr2=0x7f3308114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308114490 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f32f400b3b0 tx=0x7f32f4009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 shutdown_connections 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32f006ec00 0x7f32f00710b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3308103420 0x7f3308113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 --2- 172.21.15.44:0/2538292167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f33081046d0 0x7f3308114490 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.401+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 >> 172.21.15.44:0/2538292167 conn(0x7f3308072640 msgr2=0x7f3308073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.402+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 shutdown_connections 2024-04-16T11:53:02.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:02.402+0000 7f3310600700 1 -- 172.21.15.44:0/2538292167 wait complete. 2024-04-16T11:53:03.033 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:02 smithi044 ceph-mon[93020]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:03.034 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:02 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2538292167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:53:03.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:02 smithi045 ceph-mon[81423]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:03.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:02 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2538292167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:53:04.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:03 smithi045 ceph-mon[81423]: pgmap v297: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:04.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:03 smithi044 ceph-mon[93020]: pgmap v297: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:05.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:04 smithi045 ceph-mon[81423]: Upgrade: Updating grafana.smithi044 2024-04-16T11:53:05.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:04 smithi045 ceph-mon[81423]: Deploying daemon grafana.smithi044 on smithi044 2024-04-16T11:53:05.178 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:04 smithi044 ceph-mon[93020]: Upgrade: Updating grafana.smithi044 2024-04-16T11:53:05.178 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:04 smithi044 ceph-mon[93020]: Deploying daemon grafana.smithi044 on smithi044 2024-04-16T11:53:06.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:05 smithi045 ceph-mon[81423]: pgmap v298: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:06.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:05 smithi044 ceph-mon[93020]: pgmap v298: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:08.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:07 smithi045 ceph-mon[81423]: pgmap v299: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:07 smithi044 ceph-mon[93020]: pgmap v299: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:10.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:09 smithi045 ceph-mon[81423]: pgmap v300: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:10.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:09 smithi044 ceph-mon[93020]: pgmap v300: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:11.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:11.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:12.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:11 smithi045 ceph-mon[81423]: pgmap v301: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:12.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:11 smithi044 ceph-mon[93020]: pgmap v301: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:13.923 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:13 smithi044 ceph-mon[93020]: pgmap v302: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:14.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:13 smithi045 ceph-mon[81423]: pgmap v302: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:16.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:15 smithi045 ceph-mon[81423]: pgmap v303: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:16.209 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:15 smithi044 ceph-mon[93020]: pgmap v303: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:18.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:17 smithi045 ceph-mon[81423]: pgmap v304: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:18.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:17 smithi044 ceph-mon[93020]: pgmap v304: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:19 smithi044 ceph-mon[93020]: pgmap v305: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:19 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:19 smithi045 ceph-mon[81423]: pgmap v305: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:19 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:22.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:21 smithi044 ceph-mon[93020]: pgmap v306: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:22.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:21 smithi045 ceph-mon[81423]: pgmap v306: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:24.218 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:23 smithi044 ceph-mon[93020]: pgmap v307: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:24.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:23 smithi045 ceph-mon[81423]: pgmap v307: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:26.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:25 smithi044 ceph-mon[93020]: pgmap v308: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:26.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:25 smithi045 ceph-mon[81423]: pgmap v308: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:27.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:27 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:27 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:27.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:27 smithi045 ceph-mon[81423]: pgmap v309: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:27.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:27 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:27.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:27 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:27.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:27 smithi044 ceph-mon[93020]: pgmap v309: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:29.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:29.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:29 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:29.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:29 smithi044 ceph-mon[93020]: pgmap v310: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:29.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:29.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:29 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:29.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:29 smithi045 ceph-mon[81423]: pgmap v310: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:32.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:31 smithi045 ceph-mon[81423]: pgmap v311: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:32.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:31 smithi044 ceph-mon[93020]: pgmap v311: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.572+0000 7f5307000700 1 -- 172.21.15.44:0/3247951846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 msgr2=0x7f53000fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.572+0000 7f5307000700 1 --2- 172.21.15.44:0/3247951846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f53000fe4b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52ec004940 tx=0x7f52ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.572+0000 7f5307000700 1 -- 172.21.15.44:0/3247951846 shutdown_connections 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.572+0000 7f5307000700 1 --2- 172.21.15.44:0/3247951846 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f53000fe4b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.572+0000 7f5307000700 1 --2- 172.21.15.44:0/3247951846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53000fce40 0x7f53000fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.572+0000 7f5307000700 1 -- 172.21.15.44:0/3247951846 >> 172.21.15.44:0/3247951846 conn(0x7f53000f8530 msgr2=0x7f53000fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.573+0000 7f5307000700 1 -- 172.21.15.44:0/3247951846 shutdown_connections 2024-04-16T11:53:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.573+0000 7f5307000700 1 -- 172.21.15.44:0/3247951846 wait complete. 2024-04-16T11:53:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.574+0000 7f5307000700 1 Processor -- start 2024-04-16T11:53:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.574+0000 7f5307000700 1 -- start start 2024-04-16T11:53:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.574+0000 7f5307000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53000fce40 0x7f530010fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.574+0000 7f5307000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f5300110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.574+0000 7f5307000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5300110780 con 0x7f53000fe040 2024-04-16T11:53:32.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.574+0000 7f5307000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5300110fd0 con 0x7f53000fce40 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.575+0000 7f52ff400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f5300110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.575+0000 7f52ff400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f5300110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35100/0 (socket says 172.21.15.44:35100) 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.575+0000 7f52ff400700 1 -- 172.21.15.44:0/817623631 learned_addr learned my addr 172.21.15.44:0/817623631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.575+0000 7f52ffe00700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53000fce40 0x7f530010fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f52ff400700 1 -- 172.21.15.44:0/817623631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53000fce40 msgr2=0x7f530010fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f52ff400700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53000fce40 0x7f530010fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f52ff400700 1 -- 172.21.15.44:0/817623631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec005040 con 0x7f53000fe040 2024-04-16T11:53:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f52ff400700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f5300110180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009150 tx=0x7f52ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52ec00e040 con 0x7f53000fe040 2024-04-16T11:53:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5300111250 con 0x7f53000fe040 2024-04-16T11:53:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.576+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52ec00bbf0 con 0x7f53000fe040 2024-04-16T11:53:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.577+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52ec01c630 con 0x7f53000fe040 2024-04-16T11:53:32.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.577+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5300111790 con 0x7f53000fe040 2024-04-16T11:53:32.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.578+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53001019d0 con 0x7f53000fe040 2024-04-16T11:53:32.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.580+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f52ec00bd60 con 0x7f53000fe040 2024-04-16T11:53:32.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.581+0000 7f52f7e00700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f52e406ec10 0x7f52e40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:32.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.581+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f52ec0909a0 con 0x7f53000fe040 2024-04-16T11:53:32.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.582+0000 7f52ffe00700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f52e406ec10 0x7f52e40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:32.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.583+0000 7f52ffe00700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f52e406ec10 0x7f52e40710c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f53000fdbb0 tx=0x7f52f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:32.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.586+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f52ec05d080 con 0x7f53000fe040 2024-04-16T11:53:32.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 650, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142163 2024-04-16T11:53:32.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 650, success, outsize: 120 2024-04-16T11:53:32.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 652, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142256 2024-04-16T11:53:32.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 652, success, outsize: 120 2024-04-16T11:53:32.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 654, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142352 2024-04-16T11:53:32.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 654, success, outsize: 120 2024-04-16T11:53:32.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 656, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142454 2024-04-16T11:53:32.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 656, success, outsize: 120 2024-04-16T11:53:32.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 658, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142553 2024-04-16T11:53:32.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 658, success, outsize: 120 2024-04-16T11:53:32.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 660, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142646 2024-04-16T11:53:32.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 660, success, outsize: 120 2024-04-16T11:53:32.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 662, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142742 2024-04-16T11:53:32.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 662, success, outsize: 120 2024-04-16T11:53:32.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 664, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142844 2024-04-16T11:53:32.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 664, success, outsize: 120 2024-04-16T11:53:32.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 666, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142943 2024-04-16T11:53:32.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 666, success, outsize: 120 2024-04-16T11:53:32.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 668, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143036 2024-04-16T11:53:32.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 668, success, outsize: 120 2024-04-16T11:53:32.800 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 670, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143132 2024-04-16T11:53:32.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 670, success, outsize: 120 2024-04-16T11:53:32.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 672, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143234 2024-04-16T11:53:32.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 672, success, outsize: 120 2024-04-16T11:53:32.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 674, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143333 2024-04-16T11:53:32.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 674, success, outsize: 120 2024-04-16T11:53:32.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.846+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5300111a70 con 0x7f52e406ec10 2024-04-16T11:53:32.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.847+0000 7f52f7e00700 1 -- 172.21.15.44:0/817623631 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7f5300111a70 con 0x7f52e406ec10 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f52e406ec10 msgr2=0x7f52e40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f52e406ec10 0x7f52e40710c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f53000fdbb0 tx=0x7f52f000b040 comp rx=0 tx=0).stop 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 msgr2=0x7f5300110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f5300110180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009150 tx=0x7f52ec009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 shutdown_connections 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f52e406ec10 0x7f52e40710c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53000fce40 0x7f530010fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 --2- 172.21.15.44:0/817623631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53000fe040 0x7f5300110180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 >> 172.21.15.44:0/817623631 conn(0x7f53000f8530 msgr2=0x7f53000f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 shutdown_connections 2024-04-16T11:53:32.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:32.851+0000 7f5307000700 1 -- 172.21.15.44:0/817623631 wait complete. 2024-04-16T11:53:32.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 676, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143428 2024-04-16T11:53:32.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 676, success, outsize: 120 2024-04-16T11:53:32.866 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:53:32.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 678, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143527 2024-04-16T11:53:32.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 678, success, outsize: 120 2024-04-16T11:53:32.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 680, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143629 2024-04-16T11:53:32.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 680, success, outsize: 120 2024-04-16T11:53:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 682, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143728 2024-04-16T11:53:32.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 682, success, outsize: 120 2024-04-16T11:53:32.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 684, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143821 2024-04-16T11:53:32.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 684, success, outsize: 120 2024-04-16T11:53:32.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 686, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143917 2024-04-16T11:53:32.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 686, success, outsize: 120 2024-04-16T11:53:32.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 688, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144019 2024-04-16T11:53:32.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 688, success, outsize: 120 2024-04-16T11:53:32.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 690, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144119 2024-04-16T11:53:32.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 690, success, outsize: 120 2024-04-16T11:53:33.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 692, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144221 2024-04-16T11:53:33.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 692, success, outsize: 120 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 -- 172.21.15.44:0/3539014410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 msgr2=0x7fb92c104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 --2- 172.21.15.44:0/3539014410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c104b20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb918004940 tx=0x7fb918009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 -- 172.21.15.44:0/3539014410 shutdown_connections 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 --2- 172.21.15.44:0/3539014410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c104b20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 --2- 172.21.15.44:0/3539014410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb92c103400 0x7fb92c103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 -- 172.21.15.44:0/3539014410 >> 172.21.15.44:0/3539014410 conn(0x7fb92c072590 msgr2=0x7fb92c0749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 -- 172.21.15.44:0/3539014410 shutdown_connections 2024-04-16T11:53:33.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.006+0000 7fb934000700 1 -- 172.21.15.44:0/3539014410 wait complete. 2024-04-16T11:53:33.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.007+0000 7fb934000700 1 Processor -- start 2024-04-16T11:53:33.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.007+0000 7fb934000700 1 -- start start 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb934000700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb92c103400 0x7fb92c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb934000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb934000700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c114b50 con 0x7fb92c1046b0 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb934000700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c1153a0 con 0x7fb92c103400 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb930c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb930c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c114550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35118/0 (socket says 172.21.15.44:35118) 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.008+0000 7fb930c00700 1 -- 172.21.15.44:0/3169842298 learned_addr learned my addr 172.21.15.44:0/3169842298 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.009+0000 7fb931600700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb92c103400 0x7fb92c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.009+0000 7fb930c00700 1 -- 172.21.15.44:0/3169842298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb92c103400 msgr2=0x7fb92c114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.009+0000 7fb930c00700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb92c103400 0x7fb92c114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.009+0000 7fb930c00700 1 -- 172.21.15.44:0/3169842298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918005040 con 0x7fb92c1046b0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.009+0000 7fb930c00700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c114550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb91800b3b0 tx=0x7fb918009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.010+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb91800e040 con 0x7fb92c1046b0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.010+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c115620 con 0x7fb92c1046b0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.010+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c115b60 con 0x7fb92c1046b0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.010+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb918004290 con 0x7fb92c1046b0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.010+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb91801c3f0 con 0x7fb92c1046b0 2024-04-16T11:53:33.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.011+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c0052e0 con 0x7fb92c1046b0 2024-04-16T11:53:33.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.012+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb91800bbf0 con 0x7fb92c1046b0 2024-04-16T11:53:33.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.014+0000 7fb929600700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb91406ec10 0x7fb9140710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.014+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fb91808fe90 con 0x7fb92c1046b0 2024-04-16T11:53:33.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.014+0000 7fb931600700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb91406ec10 0x7fb9140710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.015+0000 7fb931600700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb91406ec10 0x7fb9140710c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb920004990 tx=0x7fb920009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:33.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.019+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb91805c570 con 0x7fb92c1046b0 2024-04-16T11:53:33.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 694, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144327 2024-04-16T11:53:33.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 694, success, outsize: 120 2024-04-16T11:53:33.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 696, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144429 2024-04-16T11:53:33.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 696, success, outsize: 120 2024-04-16T11:53:33.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 698, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144528 2024-04-16T11:53:33.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 698, success, outsize: 120 2024-04-16T11:53:33.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 700, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144621 2024-04-16T11:53:33.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 700, success, outsize: 120 2024-04-16T11:53:33.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 702, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144717 2024-04-16T11:53:33.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 702, success, outsize: 120 2024-04-16T11:53:33.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:unique: 704, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144819 2024-04-16T11:53:33.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr: unique: 704, success, outsize: 120 2024-04-16T11:53:33.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.272+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb90c000bf0 con 0x7fb91406ec10 2024-04-16T11:53:33.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.275+0000 7fb929600700 1 -- 172.21.15.44:0/3169842298 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7fb90c000bf0 con 0x7fb91406ec10 2024-04-16T11:53:33.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.277+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb91406ec10 msgr2=0x7fb9140710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.277+0000 7fb934000700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb91406ec10 0x7fb9140710c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb920004990 tx=0x7fb920009ee0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.277+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 msgr2=0x7fb92c114550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.277+0000 7fb934000700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c114550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb91800b3b0 tx=0x7fb918009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 shutdown_connections 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb91406ec10 0x7fb9140710c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb92c103400 0x7fb92c114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 --2- 172.21.15.44:0/3169842298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb92c1046b0 0x7fb92c114550 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 >> 172.21.15.44:0/3169842298 conn(0x7fb92c072590 msgr2=0x7fb92c073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 shutdown_connections 2024-04-16T11:53:33.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.278+0000 7fb934000700 1 -- 172.21.15.44:0/3169842298 wait complete. 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 -- 172.21.15.44:0/1022788769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 msgr2=0x7fac34104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1022788769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34104970 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fac20004940 tx=0x7fac20009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 -- 172.21.15.44:0/1022788769 shutdown_connections 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1022788769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34104970 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1022788769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac34103250 0x7fac34103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 -- 172.21.15.44:0/1022788769 >> 172.21.15.44:0/1022788769 conn(0x7fac34072500 msgr2=0x7fac34074950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.429+0000 7fac3ae00700 1 -- 172.21.15.44:0/1022788769 shutdown_connections 2024-04-16T11:53:33.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.430+0000 7fac3ae00700 1 -- 172.21.15.44:0/1022788769 wait complete. 2024-04-16T11:53:33.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.430+0000 7fac3ae00700 1 Processor -- start 2024-04-16T11:53:33.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac3ae00700 1 -- start start 2024-04-16T11:53:33.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac3ae00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac34103250 0x7fac34113ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac3ae00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34114430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac3ae00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34114a30 con 0x7fac34104500 2024-04-16T11:53:33.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac3ae00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34115170 con 0x7fac34103250 2024-04-16T11:53:33.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac39000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34114430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac39000700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34114430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35142/0 (socket says 172.21.15.44:35142) 2024-04-16T11:53:33.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.431+0000 7fac39000700 1 -- 172.21.15.44:0/1774886153 learned_addr learned my addr 172.21.15.44:0/1774886153 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:33.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac39000700 1 -- 172.21.15.44:0/1774886153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac34103250 msgr2=0x7fac34113ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac39a00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac34103250 0x7fac34113ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac39000700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac34103250 0x7fac34113ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac39000700 1 -- 172.21.15.44:0/1774886153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac20005040 con 0x7fac34104500 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac39000700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34114430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac2000b3b0 tx=0x7fac20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac2000e040 con 0x7fac34104500 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac20004290 con 0x7fac34104500 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac2001c630 con 0x7fac34104500 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.432+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac341153f0 con 0x7fac34104500 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.433+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac341159b0 con 0x7fac34104500 2024-04-16T11:53:33.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.434+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3405f5f0 con 0x7fac34104500 2024-04-16T11:53:33.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.435+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fac2000bbf0 con 0x7fac34104500 2024-04-16T11:53:33.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.435+0000 7fac31600700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fac1c06ec10 0x7fac1c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.436+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fac2008ffa0 con 0x7fac34104500 2024-04-16T11:53:33.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.436+0000 7fac39a00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fac1c06ec10 0x7fac1c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.436+0000 7fac39a00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fac1c06ec10 0x7fac1c0710c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac28004c40 tx=0x7fac28009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:33.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.442+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fac2005c600 con 0x7fac34104500 2024-04-16T11:53:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.679+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fac34115ce0 con 0x7fac1c06ec10 2024-04-16T11:53:33.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.696+0000 7fac31600700 1 -- 172.21.15.44:0/1774886153 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fac34115ce0 con 0x7fac1c06ec10 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (48s) 7s ago 21m 14.2M - 0.25.0 c8568f914cd2 42392ee6c260 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (7m) 7s ago 21m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (7m) 96s ago 20m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (14s) 7s ago 20m 50.6M - 9.4.12 2bacad6d85d8 29f522faff8a 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (3m) 7s ago 13m 12.1M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (3m) 7s ago 14m 21.8M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:53:33.694 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (2m) 96s ago 13m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 e08aedf9cf6c 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (2m) 96s ago 13m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (10m) 7s ago 22m 520M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (9m) 96s ago 19m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (9m) 7s ago 23m 56.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (8m) 96s ago 19m 39.8M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (106s) 7s ago 21m 7377k - 1.5.0 0da6a335fe13 d6c7c8a1653a 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (100s) 96s ago 19m 3007k - 1.5.0 0da6a335fe13 e9596e00b617 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (6m) 7s ago 18m 55.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 7s ago 18m 46.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (5m) 7s ago 17m 45.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (5m) 96s ago 16m 45.2M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (4m) 96s ago 16m 42.6M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:53:33.695 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (4m) 96s ago 15m 49.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:53:33.696 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (74s) 7s ago 20m 48.6M - 2.43.0 a07b618ecd1d b3776ca300db 2024-04-16T11:53:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fac1c06ec10 msgr2=0x7fac1c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fac1c06ec10 0x7fac1c0710c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac28004c40 tx=0x7fac28009040 comp rx=0 tx=0).stop 2024-04-16T11:53:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 msgr2=0x7fac34114430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34114430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac2000b3b0 tx=0x7fac20009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 shutdown_connections 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fac1c06ec10 0x7fac1c0710c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac34103250 0x7fac34113ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 --2- 172.21.15.44:0/1774886153 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac34104500 0x7fac34114430 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 >> 172.21.15.44:0/1774886153 conn(0x7fac34072500 msgr2=0x7fac34074b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.699+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 shutdown_connections 2024-04-16T11:53:33.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.700+0000 7fac3ae00700 1 -- 172.21.15.44:0/1774886153 wait complete. 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 -- 172.21.15.44:0/3633143798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 msgr2=0x7fb3380fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 --2- 172.21.15.44:0/3633143798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb3380fe350 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb324004940 tx=0x7fb324009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 -- 172.21.15.44:0/3633143798 shutdown_connections 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 --2- 172.21.15.44:0/3633143798 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb3380fe350 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 --2- 172.21.15.44:0/3633143798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3380fcc30 0x7fb3380fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 -- 172.21.15.44:0/3633143798 >> 172.21.15.44:0/3633143798 conn(0x7fb3380f83d0 msgr2=0x7fb3380fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 -- 172.21.15.44:0/3633143798 shutdown_connections 2024-04-16T11:53:33.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.846+0000 7fb33e400700 1 -- 172.21.15.44:0/3633143798 wait complete. 2024-04-16T11:53:33.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.847+0000 7fb33e400700 1 Processor -- start 2024-04-16T11:53:33.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb33e400700 1 -- start start 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb33e400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3380fcc30 0x7fb33810fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb33e400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb33810ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb33e400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338110540 con 0x7fb3380fdee0 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb33e400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338110d90 con 0x7fb3380fcc30 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb337e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb33810ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb337e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb33810ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35162/0 (socket says 172.21.15.44:35162) 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.848+0000 7fb337e00700 1 -- 172.21.15.44:0/2246816431 learned_addr learned my addr 172.21.15.44:0/2246816431 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:33.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.849+0000 7fb33d000700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3380fcc30 0x7fb33810fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.849+0000 7fb337e00700 1 -- 172.21.15.44:0/2246816431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3380fcc30 msgr2=0x7fb33810fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.849+0000 7fb337e00700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3380fcc30 0x7fb33810fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.849+0000 7fb337e00700 1 -- 172.21.15.44:0/2246816431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb324005040 con 0x7fb3380fdee0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.849+0000 7fb337e00700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb33810ff40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb324013870 tx=0x7fb324009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.849+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb32400e070 con 0x7fb3380fdee0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.850+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb32401c400 con 0x7fb3380fdee0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.850+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb32400b630 con 0x7fb3380fdee0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.850+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb338111010 con 0x7fb3380fdee0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.850+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3381114f0 con 0x7fb3380fdee0 2024-04-16T11:53:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.851+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb338101870 con 0x7fb3380fdee0 2024-04-16T11:53:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.852+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb32401c570 con 0x7fb3380fdee0 2024-04-16T11:53:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.853+0000 7fb334c00700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb32006ec10 0x7fb3200710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:33.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.853+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fb324090230 con 0x7fb3380fdee0 2024-04-16T11:53:33.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.853+0000 7fb33d000700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb32006ec10 0x7fb3200710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:33.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.854+0000 7fb33d000700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb32006ec10 0x7fb3200710c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb32c004600 tx=0x7fb32c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:33.858+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb32405c910 con 0x7fb3380fdee0 2024-04-16T11:53:33.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:33 smithi044 ceph-mon[93020]: pgmap v312: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:33.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:33 smithi044 ceph-mon[93020]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:34.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:33 smithi045 ceph-mon[81423]: pgmap v312: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:34.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:33 smithi045 ceph-mon[81423]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:34.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.174+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb33804bed0 con 0x7fb3380fdee0 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.175+0000 7fb334c00700 1 -- 172.21.15.44:0/2246816431 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb32405c4a0 con 0x7fb3380fdee0 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:53:34.173 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:53:34.174 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T11:53:34.175 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:53:34.175 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:53:34.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb32006ec10 msgr2=0x7fb3200710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb32006ec10 0x7fb3200710c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb32c004600 tx=0x7fb32c00b040 comp rx=0 tx=0).stop 2024-04-16T11:53:34.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 msgr2=0x7fb33810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb33810ff40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb324013870 tx=0x7fb324009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 shutdown_connections 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fb32006ec10 0x7fb3200710c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3380fcc30 0x7fb33810fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 --2- 172.21.15.44:0/2246816431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3380fdee0 0x7fb33810ff40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.178+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 >> 172.21.15.44:0/2246816431 conn(0x7fb3380f83d0 msgr2=0x7fb3380f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.179+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 shutdown_connections 2024-04-16T11:53:34.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.179+0000 7fb33e400700 1 -- 172.21.15.44:0/2246816431 wait complete. 2024-04-16T11:53:34.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.329+0000 7f38f6800700 1 -- 172.21.15.44:0/3751043824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 msgr2=0x7f38f0104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.329+0000 7f38f6800700 1 --2- 172.21.15.44:0/3751043824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0104750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f38d8004940 tx=0x7f38d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.329+0000 7f38f6800700 1 -- 172.21.15.44:0/3751043824 shutdown_connections 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.329+0000 7f38f6800700 1 --2- 172.21.15.44:0/3751043824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0104750 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.329+0000 7f38f6800700 1 --2- 172.21.15.44:0/3751043824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f00662f0 0x7f38f0101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.329+0000 7f38f6800700 1 -- 172.21.15.44:0/3751043824 >> 172.21.15.44:0/3751043824 conn(0x7f38f00f74c0 msgr2=0x7f38f00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.330+0000 7f38f6800700 1 -- 172.21.15.44:0/3751043824 shutdown_connections 2024-04-16T11:53:34.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.330+0000 7f38f6800700 1 -- 172.21.15.44:0/3751043824 wait complete. 2024-04-16T11:53:34.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.331+0000 7f38f6800700 1 Processor -- start 2024-04-16T11:53:34.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.331+0000 7f38f6800700 1 -- start start 2024-04-16T11:53:34.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38f6800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f00662f0 0x7f38f010fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:34.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38f6800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:34.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38f6800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0110730 con 0x7f38f0102370 2024-04-16T11:53:34.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38f6800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0110f30 con 0x7f38f00662f0 2024-04-16T11:53:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38e6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38e6a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0110130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35184/0 (socket says 172.21.15.44:35184) 2024-04-16T11:53:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38e6a00700 1 -- 172.21.15.44:0/2006594870 learned_addr learned my addr 172.21.15.44:0/2006594870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:34.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.332+0000 7f38ef400700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f00662f0 0x7f38f010fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e6a00700 1 -- 172.21.15.44:0/2006594870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f00662f0 msgr2=0x7f38f010fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e6a00700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f00662f0 0x7f38f010fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e6a00700 1 -- 172.21.15.44:0/2006594870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d8005040 con 0x7f38f0102370 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e6a00700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0110130 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f38d800b0d0 tx=0x7f38d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38d800e070 con 0x7f38f0102370 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38d800b410 con 0x7f38f0102370 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38d80125e0 con 0x7f38f0102370 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.333+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f01111b0 con 0x7f38f0102370 2024-04-16T11:53:34.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.334+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f01116f0 con 0x7f38f0102370 2024-04-16T11:53:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.335+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f005f5f0 con 0x7f38f0102370 2024-04-16T11:53:34.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.336+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f38d800b580 con 0x7f38f0102370 2024-04-16T11:53:34.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.337+0000 7f38e7e00700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38d406ec10 0x7f38d40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:34.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.338+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f38d808f830 con 0x7f38f0102370 2024-04-16T11:53:34.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.338+0000 7f38ef400700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38d406ec10 0x7f38d40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:34.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.339+0000 7f38ef400700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38d406ec10 0x7f38d40710c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f38f0066030 tx=0x7f38e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:34.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.343+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f38d805bf10 con 0x7f38f0102370 2024-04-16T11:53:34.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.489+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 7 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38d800b280 con 0x7f38f0102370 2024-04-16T11:53:34.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.653+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f38f01119b0 con 0x7f38f0102370 2024-04-16T11:53:34.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.654+0000 7f38e7e00700 1 -- 172.21.15.44:0/2006594870 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 37 v37) v1 ==== 76+0+1932 (secure 0 0 0) 0x7f38d8022020 con 0x7f38f0102370 2024-04-16T11:53:34.652 INFO:teuthology.orchestra.run.smithi044.stdout:e37 2024-04-16T11:53:34.652 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:53:34.652 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 36 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:51:22.505053+0000 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:53:34.653 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 91 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:53:34.654 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=34432} 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 34432 members: 34432 2024-04-16T11:53:34.655 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:34432} state up:active seq 24 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:44319} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:34.656 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:44341} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:53:34.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38d406ec10 msgr2=0x7f38d40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38d406ec10 0x7f38d40710c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f38f0066030 tx=0x7f38e000b040 comp rx=0 tx=0).stop 2024-04-16T11:53:34.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 msgr2=0x7f38f0110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0110130 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f38d800b0d0 tx=0x7f38d8009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 shutdown_connections 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f38d406ec10 0x7f38d40710c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38f00662f0 0x7f38f010fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 --2- 172.21.15.44:0/2006594870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38f0102370 0x7f38f0110130 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 >> 172.21.15.44:0/2006594870 conn(0x7f38f00f74c0 msgr2=0x7f38f00f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 shutdown_connections 2024-04-16T11:53:34.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.658+0000 7f38f6800700 1 -- 172.21.15.44:0/2006594870 wait complete. 2024-04-16T11:53:34.659 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.815+0000 7fe671200700 1 -- 172.21.15.44:0/1282057389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c073170 msgr2=0x7fe66c073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.815+0000 7fe671200700 1 --2- 172.21.15.44:0/1282057389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c073170 0x7fe66c073b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe658004940 tx=0x7fe658009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.815+0000 7fe671200700 1 -- 172.21.15.44:0/1282057389 shutdown_connections 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.815+0000 7fe671200700 1 --2- 172.21.15.44:0/1282057389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c073170 0x7fe66c073b80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.815+0000 7fe671200700 1 --2- 172.21.15.44:0/1282057389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66c072820 0x7fe66c072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.815+0000 7fe671200700 1 -- 172.21.15.44:0/1282057389 >> 172.21.15.44:0/1282057389 conn(0x7fe66c0fa790 msgr2=0x7fe66c0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.816+0000 7fe671200700 1 -- 172.21.15.44:0/1282057389 shutdown_connections 2024-04-16T11:53:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.816+0000 7fe671200700 1 -- 172.21.15.44:0/1282057389 wait complete. 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.816+0000 7fe671200700 1 Processor -- start 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe671200700 1 -- start start 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe671200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 0x7fe66c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe671200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66c114490 0x7fe66c1194c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe671200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c114950 con 0x7fe66c072820 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe671200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c114a90 con 0x7fe66c114490 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe669e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 0x7fe66c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe669e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 0x7fe66c113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35196/0 (socket says 172.21.15.44:35196) 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe669e00700 1 -- 172.21.15.44:0/2097355805 learned_addr learned my addr 172.21.15.44:0/2097355805 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.817+0000 7fe669400700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66c114490 0x7fe66c1194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:34.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe669e00700 1 -- 172.21.15.44:0/2097355805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66c114490 msgr2=0x7fe66c1194c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:34.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe669e00700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66c114490 0x7fe66c1194c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:34.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe669e00700 1 -- 172.21.15.44:0/2097355805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658005040 con 0x7fe66c072820 2024-04-16T11:53:34.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe669e00700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 0x7fe66c113f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe65c002a50 tx=0x7fe65c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:34.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65c007cc0 con 0x7fe66c072820 2024-04-16T11:53:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe65c00f040 con 0x7fe66c072820 2024-04-16T11:53:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65c00da40 con 0x7fe66c072820 2024-04-16T11:53:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.818+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c119a60 con 0x7fe66c072820 2024-04-16T11:53:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.819+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c119fd0 con 0x7fe66c072820 2024-04-16T11:53:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.820+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c04c710 con 0x7fe66c072820 2024-04-16T11:53:34.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.821+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe65c0097c0 con 0x7fe66c072820 2024-04-16T11:53:34.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.822+0000 7fe662000700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe65006ec00 0x7fe6500710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:34.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.822+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fe65c08e820 con 0x7fe66c072820 2024-04-16T11:53:34.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.822+0000 7fe669400700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe65006ec00 0x7fe6500710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:34.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.823+0000 7fe669400700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe65006ec00 0x7fe6500710b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe658004f90 tx=0x7fe658009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:34.827+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe65c05ae80 con 0x7fe66c072820 2024-04-16T11:53:34.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:34.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:34.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2246816431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.978 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.979 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T11:53:34.980 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:34.981 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T11:53:34.982 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T11:53:34.982 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:34.982 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:34 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2006594870' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:53:35.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.076+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe66c10a480 con 0x7fe65006ec00 2024-04-16T11:53:35.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.078+0000 7fe662000700 1 -- 172.21.15.44:0/2097355805 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fe66c10a480 con 0x7fe65006ec00 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": null, 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": false, 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "", 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": null, 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:53:35.076 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:53:35.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe65006ec00 msgr2=0x7fe6500710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe65006ec00 0x7fe6500710b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe658004f90 tx=0x7fe658009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 msgr2=0x7fe66c113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 0x7fe66c113f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe65c002a50 tx=0x7fe65c004600 comp rx=0 tx=0).stop 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 shutdown_connections 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe65006ec00 0x7fe6500710b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe66c072820 0x7fe66c113f50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 --2- 172.21.15.44:0/2097355805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66c114490 0x7fe66c1194c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.080+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 >> 172.21.15.44:0/2097355805 conn(0x7fe66c0fa790 msgr2=0x7fe66c0fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.081+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 shutdown_connections 2024-04-16T11:53:35.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.081+0000 7fe671200700 1 -- 172.21.15.44:0/2097355805 wait complete. 2024-04-16T11:53:35.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:35.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2246816431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.081 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.082 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.083 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T11:53:35.084 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T11:53:35.085 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T11:53:35.086 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T11:53:35.087 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T11:53:35.087 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:35.087 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:34 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2006594870' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 -- 172.21.15.44:0/3616239078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fce40 msgr2=0x7f621c0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 --2- 172.21.15.44:0/3616239078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fce40 0x7f621c0fd250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6210004940 tx=0x7f6210009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 -- 172.21.15.44:0/3616239078 shutdown_connections 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 --2- 172.21.15.44:0/3616239078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c0fe040 0x7f621c0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 --2- 172.21.15.44:0/3616239078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fce40 0x7f621c0fd250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 -- 172.21.15.44:0/3616239078 >> 172.21.15.44:0/3616239078 conn(0x7f621c0f8530 msgr2=0x7f621c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 -- 172.21.15.44:0/3616239078 shutdown_connections 2024-04-16T11:53:35.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.230+0000 7f6223a00700 1 -- 172.21.15.44:0/3616239078 wait complete. 2024-04-16T11:53:35.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.231+0000 7f6223a00700 1 Processor -- start 2024-04-16T11:53:35.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.231+0000 7f6223a00700 1 -- start start 2024-04-16T11:53:35.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.231+0000 7f6223a00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c0fce40 0x7f621c10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.231+0000 7f6223a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 0x7f621c10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.231+0000 7f6223a00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c10e5d0 con 0x7f621c0fe040 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.231+0000 7f6223a00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c10ee20 con 0x7f621c0fce40 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f621be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 0x7f621c10dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f621be00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 0x7f621c10dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35200/0 (socket says 172.21.15.44:35200) 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f621be00700 1 -- 172.21.15.44:0/4045245296 learned_addr learned my addr 172.21.15.44:0/4045245296 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f6221000700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c0fce40 0x7f621c10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f621be00700 1 -- 172.21.15.44:0/4045245296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c0fce40 msgr2=0x7f621c10da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f621be00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c0fce40 0x7f621c10da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.232+0000 7f621be00700 1 -- 172.21.15.44:0/4045245296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6210005040 con 0x7f621c0fe040 2024-04-16T11:53:35.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.233+0000 7f621be00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 0x7f621c10dfd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6208002a50 tx=0x7f62080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:35.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.233+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6208007cc0 con 0x7f621c0fe040 2024-04-16T11:53:35.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.233+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f620800f040 con 0x7f621c0fe040 2024-04-16T11:53:35.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.233+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6208009640 con 0x7f621c0fe040 2024-04-16T11:53:35.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.233+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621c10f100 con 0x7f621c0fe040 2024-04-16T11:53:35.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.233+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621c10f640 con 0x7f621c0fe040 2024-04-16T11:53:35.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.234+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621c05f5f0 con 0x7f621c0fe040 2024-04-16T11:53:35.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.235+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f62080097a0 con 0x7f621c0fe040 2024-04-16T11:53:35.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.237+0000 7f6218c00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f620406ec00 0x7f62040710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:53:35.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.237+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f620808e810 con 0x7f621c0fe040 2024-04-16T11:53:35.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.237+0000 7f6221000700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f620406ec00 0x7f62040710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:53:35.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.238+0000 7f6221000700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f620406ec00 0x7f62040710b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f621000b970 tx=0x7f6210009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:53:35.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.241+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f620805af10 con 0x7f621c0fe040 2024-04-16T11:53:35.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.562+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f621c04bed0 con 0x7f621c0fe040 2024-04-16T11:53:35.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.563+0000 7f6218c00700 1 -- 172.21.15.44:0/4045245296 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f620805aaa0 con 0x7f621c0fe040 2024-04-16T11:53:35.561 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f620406ec00 msgr2=0x7f62040710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f620406ec00 0x7f62040710b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f621000b970 tx=0x7f6210009e30 comp rx=0 tx=0).stop 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 msgr2=0x7f621c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 0x7f621c10dfd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6208002a50 tx=0x7f62080046b0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 shutdown_connections 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f620406ec00 0x7f62040710b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f621c0fce40 0x7f621c10da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 --2- 172.21.15.44:0/4045245296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f621c0fe040 0x7f621c10dfd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:53:35.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 >> 172.21.15.44:0/4045245296 conn(0x7f621c0f8530 msgr2=0x7f621c0f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:53:35.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 shutdown_connections 2024-04-16T11:53:35.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:53:35.566+0000 7f6223a00700 1 -- 172.21.15.44:0/4045245296 wait complete. 2024-04-16T11:53:35.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: Upgrade: Finalizing container_image settings 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: Upgrade: Complete! 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: pgmap v313: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:35.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/4045245296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:53:35.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:35 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T11:53:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: Upgrade: Finalizing container_image settings 2024-04-16T11:53:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: Upgrade: Complete! 2024-04-16T11:53:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: pgmap v313: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/4045245296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:53:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:35 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:36 smithi045 ceph-mon[81423]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:37.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:36 smithi044 ceph-mon[93020]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:53:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:53:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:53:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:53:37.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:53:38.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:37 smithi044 ceph-mon[93020]: pgmap v314: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:38.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:37 smithi045 ceph-mon[81423]: pgmap v314: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:40.132 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:39 smithi045 ceph-mon[81423]: pgmap v315: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:40.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:39 smithi044 ceph-mon[93020]: pgmap v315: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:42.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:41 smithi044 ceph-mon[93020]: pgmap v316: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:42.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:41 smithi045 ceph-mon[81423]: pgmap v316: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:43 smithi044 ceph-mon[93020]: pgmap v317: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:44.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:43 smithi045 ceph-mon[81423]: pgmap v317: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:46.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:45 smithi044 ceph-mon[93020]: pgmap v318: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:46.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:45 smithi045 ceph-mon[81423]: pgmap v318: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:48.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:47 smithi044 ceph-mon[93020]: pgmap v319: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:47 smithi045 ceph-mon[81423]: pgmap v319: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:49 smithi044 ceph-mon[93020]: pgmap v320: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:50.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:49 smithi045 ceph-mon[81423]: pgmap v320: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:52.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:51 smithi044 ceph-mon[93020]: pgmap v321: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:51 smithi045 ceph-mon[81423]: pgmap v321: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:54.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:53 smithi044 ceph-mon[93020]: pgmap v322: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:54.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:53 smithi045 ceph-mon[81423]: pgmap v322: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:55 smithi044 ceph-mon[93020]: pgmap v323: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:55 smithi045 ceph-mon[81423]: pgmap v323: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:53:58.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:57 smithi044 ceph-mon[93020]: pgmap v324: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:53:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:57 smithi045 ceph-mon[81423]: pgmap v324: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:00.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:53:59 smithi044 ceph-mon[93020]: pgmap v325: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:53:59 smithi045 ceph-mon[81423]: pgmap v325: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:01 smithi044 ceph-mon[93020]: pgmap v326: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:01 smithi045 ceph-mon[81423]: pgmap v326: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:04.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:03 smithi044 ceph-mon[93020]: pgmap v327: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:03 smithi045 ceph-mon[81423]: pgmap v327: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 -- 172.21.15.44:0/3168194725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9781046d0 msgr2=0x7ff978104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 --2- 172.21.15.44:0/3168194725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9781046d0 0x7ff978104b40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff964004940 tx=0x7ff964009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 -- 172.21.15.44:0/3168194725 shutdown_connections 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 --2- 172.21.15.44:0/3168194725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9781046d0 0x7ff978104b40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 --2- 172.21.15.44:0/3168194725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff978103420 0x7ff978103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 -- 172.21.15.44:0/3168194725 >> 172.21.15.44:0/3168194725 conn(0x7ff978072640 msgr2=0x7ff978074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.739+0000 7ff97f400700 1 -- 172.21.15.44:0/3168194725 shutdown_connections 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.740+0000 7ff97f400700 1 -- 172.21.15.44:0/3168194725 wait complete. 2024-04-16T11:54:05.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.741+0000 7ff97f400700 1 Processor -- start 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.741+0000 7ff97f400700 1 -- start start 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.741+0000 7ff97f400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 0x7ff978113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff97f400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9781046d0 0x7ff9781144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff97f400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978114aa0 con 0x7ff978103420 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff97f400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978114c10 con 0x7ff9781046d0 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff97ca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 0x7ff978113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff97ca00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 0x7ff978113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54224/0 (socket says 172.21.15.44:54224) 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff97ca00700 1 -- 172.21.15.44:0/4016963809 learned_addr learned my addr 172.21.15.44:0/4016963809 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:54:05.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.742+0000 7ff977e00700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9781046d0 0x7ff9781144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:05.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff97ca00700 1 -- 172.21.15.44:0/4016963809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9781046d0 msgr2=0x7ff9781144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:05.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff97ca00700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9781046d0 0x7ff9781144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:05.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff97ca00700 1 -- 172.21.15.44:0/4016963809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff964005040 con 0x7ff978103420 2024-04-16T11:54:05.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff97ca00700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 0x7ff978113f60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00c8c0 tx=0x7ff96c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff96c00a760 con 0x7ff978103420 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff96c00ad90 con 0x7ff978103420 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff978119670 con 0x7ff978103420 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.743+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff96c0113a0 con 0x7ff978103420 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.744+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff978119c30 con 0x7ff978103420 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.745+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97804c710 con 0x7ff978103420 2024-04-16T11:54:05.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.746+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff96c012400 con 0x7ff978103420 2024-04-16T11:54:05.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.747+0000 7ff974c00700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96006ec00 0x7ff9600710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:05.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.748+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7ff96c08ebc0 con 0x7ff978103420 2024-04-16T11:54:05.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.748+0000 7ff977e00700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96006ec00 0x7ff9600710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:05.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.749+0000 7ff977e00700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96006ec00 0x7ff9600710b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff96400b3b0 tx=0x7ff964009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:05.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:05.753+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff96c05b220 con 0x7ff978103420 2024-04-16T11:54:06.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.005+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff978108a30 con 0x7ff96006ec00 2024-04-16T11:54:06.003 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:05 smithi044 ceph-mon[93020]: pgmap v328: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:06.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.007+0000 7ff974c00700 1 -- 172.21.15.44:0/4016963809 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7ff978108a30 con 0x7ff96006ec00 2024-04-16T11:54:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.010+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96006ec00 msgr2=0x7ff9600710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.010+0000 7ff97f400700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96006ec00 0x7ff9600710b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff96400b3b0 tx=0x7ff964009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.010+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 msgr2=0x7ff978113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.010+0000 7ff97f400700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 0x7ff978113f60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00c8c0 tx=0x7ff96c0044c0 comp rx=0 tx=0).stop 2024-04-16T11:54:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 shutdown_connections 2024-04-16T11:54:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff96006ec00 0x7ff9600710b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff978103420 0x7ff978113f60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 --2- 172.21.15.44:0/4016963809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9781046d0 0x7ff9781144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 >> 172.21.15.44:0/4016963809 conn(0x7ff978072640 msgr2=0x7ff978073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 shutdown_connections 2024-04-16T11:54:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:06.011+0000 7ff97f400700 1 -- 172.21.15.44:0/4016963809 wait complete. 2024-04-16T11:54:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:05 smithi045 ceph-mon[81423]: pgmap v328: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:06.826 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch ps' 2024-04-16T11:54:07.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:06 smithi044 ceph-mon[93020]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:54:07.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:06 smithi045 ceph-mon[81423]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:54:07.407 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:54:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:07 smithi044 ceph-mon[93020]: pgmap v329: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:07 smithi045 ceph-mon[81423]: pgmap v329: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.502+0000 7f0b61200700 1 -- 172.21.15.44:0/1963453142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c1037c0 msgr2=0x7f0b5c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.502+0000 7f0b61200700 1 --2- 172.21.15.44:0/1963453142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c1037c0 0x7f0b5c0fce30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0b44004940 tx=0x7f0b44009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.503+0000 7f0b61200700 1 -- 172.21.15.44:0/1963453142 shutdown_connections 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.503+0000 7f0b61200700 1 --2- 172.21.15.44:0/1963453142 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c1037c0 0x7f0b5c0fce30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.503+0000 7f0b61200700 1 --2- 172.21.15.44:0/1963453142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b5c102e20 0x7f0b5c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.503+0000 7f0b61200700 1 -- 172.21.15.44:0/1963453142 >> 172.21.15.44:0/1963453142 conn(0x7f0b5c0f89d0 msgr2=0x7f0b5c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.503+0000 7f0b61200700 1 -- 172.21.15.44:0/1963453142 shutdown_connections 2024-04-16T11:54:09.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.503+0000 7f0b61200700 1 -- 172.21.15.44:0/1963453142 wait complete. 2024-04-16T11:54:09.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.504+0000 7f0b61200700 1 Processor -- start 2024-04-16T11:54:09.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.504+0000 7f0b61200700 1 -- start start 2024-04-16T11:54:09.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.504+0000 7f0b61200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 0x7f0b5c115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:09.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.504+0000 7f0b61200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b5c1037c0 0x7f0b5c1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:09.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.504+0000 7f0b61200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c116930 con 0x7f0b5c102e20 2024-04-16T11:54:09.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.504+0000 7f0b61200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10fd00 con 0x7f0b5c1037c0 2024-04-16T11:54:09.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 0x7f0b5c115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:09.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 0x7f0b5c115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54236/0 (socket says 172.21.15.44:54236) 2024-04-16T11:54:09.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 -- 172.21.15.44:0/2987458983 learned_addr learned my addr 172.21.15.44:0/2987458983 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:54:09.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 -- 172.21.15.44:0/2987458983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b5c1037c0 msgr2=0x7f0b5c1162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b51400700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b5c1037c0 0x7f0b5c1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b5c1037c0 0x7f0b5c1162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 -- 172.21.15.44:0/2987458983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b44005040 con 0x7f0b5c102e20 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.505+0000 7f0b59e00700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 0x7f0b5c115da0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00c830 tx=0x7f0b4c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.506+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b4c00a760 con 0x7f0b5c102e20 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.506+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c10ffe0 con 0x7f0b5c102e20 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.506+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b4c00ad90 con 0x7f0b5c102e20 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.506+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b4c0113a0 con 0x7f0b5c102e20 2024-04-16T11:54:09.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.506+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c1105a0 con 0x7f0b5c102e20 2024-04-16T11:54:09.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.507+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c101830 con 0x7f0b5c102e20 2024-04-16T11:54:09.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.508+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b4c012400 con 0x7f0b5c102e20 2024-04-16T11:54:09.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.509+0000 7f0b52a00700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b4006ec50 0x7f0b40071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:09.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.509+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f0b4c08ebf0 con 0x7f0b5c102e20 2024-04-16T11:54:09.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.510+0000 7f0b51400700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b4006ec50 0x7f0b40071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:09.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.510+0000 7f0b51400700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b4006ec50 0x7f0b40071100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000c00 tx=0x7f0b44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:09.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.514+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b4c05b250 con 0x7f0b5c102e20 2024-04-16T11:54:09.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.757+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b5c109f70 con 0x7f0b4006ec50 2024-04-16T11:54:09.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.787+0000 7f0b52a00700 1 -- 172.21.15.44:0/2987458983 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f0b5c109f70 con 0x7f0b4006ec50 2024-04-16T11:54:09.784 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (84s) 43s ago 21m 14.2M - 0.25.0 c8568f914cd2 42392ee6c260 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (8m) 43s ago 22m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (7m) 2m ago 20m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (50s) 43s ago 21m 50.6M - 9.4.12 2bacad6d85d8 29f522faff8a 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (3m) 43s ago 14m 12.1M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (4m) 43s ago 14m 21.8M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (2m) 2m ago 14m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 e08aedf9cf6c 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (3m) 2m ago 14m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (10m) 43s ago 23m 520M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:54:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (10m) 2m ago 20m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (9m) 43s ago 23m 56.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (9m) 2m ago 20m 39.8M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (2m) 43s ago 21m 7377k - 1.5.0 0da6a335fe13 d6c7c8a1653a 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (2m) 2m ago 20m 3007k - 1.5.0 0da6a335fe13 e9596e00b617 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (7m) 43s ago 19m 55.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 43s ago 18m 46.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (6m) 43s ago 18m 45.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (5m) 2m ago 17m 45.2M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (5m) 2m ago 16m 42.6M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:54:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (4m) 2m ago 16m 49.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:54:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (110s) 43s ago 21m 48.6M - 2.43.0 a07b618ecd1d b3776ca300db 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.790+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b4006ec50 msgr2=0x7f0b40071100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.790+0000 7f0b61200700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b4006ec50 0x7f0b40071100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000c00 tx=0x7f0b44009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 msgr2=0x7f0b5c115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 0x7f0b5c115da0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00c830 tx=0x7f0b4c004470 comp rx=0 tx=0).stop 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 shutdown_connections 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0b4006ec50 0x7f0b40071100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:09.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b5c102e20 0x7f0b5c115da0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 --2- 172.21.15.44:0/2987458983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b5c1037c0 0x7f0b5c1162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 >> 172.21.15.44:0/2987458983 conn(0x7f0b5c0f89d0 msgr2=0x7f0b5c1021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 shutdown_connections 2024-04-16T11:54:09.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:09.791+0000 7f0b61200700 1 -- 172.21.15.44:0/2987458983 wait complete. 2024-04-16T11:54:10.174 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:09 smithi044 ceph-mon[93020]: pgmap v330: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:10.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:09 smithi045 ceph-mon[81423]: pgmap v330: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:10.618 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch upgrade status' 2024-04-16T11:54:11.158 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:54:11.316 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:10 smithi044 ceph-mon[93020]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:54:11.316 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:11 smithi045 ceph-mon[81423]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:54:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:12.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:12 smithi045 ceph-mon[81423]: pgmap v331: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:12.343 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:12 smithi044 ceph-mon[93020]: pgmap v331: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:13.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 -- 172.21.15.44:0/4045344689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c102e20 msgr2=0x7fd04c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:13.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 --2- 172.21.15.44:0/4045344689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c102e20 0x7fd04c1031f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd03c004940 tx=0x7fd03c009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:13.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 -- 172.21.15.44:0/4045344689 shutdown_connections 2024-04-16T11:54:13.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 --2- 172.21.15.44:0/4045344689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd04c1037c0 0x7fd04c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 --2- 172.21.15.44:0/4045344689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c102e20 0x7fd04c1031f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 -- 172.21.15.44:0/4045344689 >> 172.21.15.44:0/4045344689 conn(0x7fd04c0f89d0 msgr2=0x7fd04c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 -- 172.21.15.44:0/4045344689 shutdown_connections 2024-04-16T11:54:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.401+0000 7fd052800700 1 -- 172.21.15.44:0/4045344689 wait complete. 2024-04-16T11:54:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.402+0000 7fd052800700 1 Processor -- start 2024-04-16T11:54:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.402+0000 7fd052800700 1 -- start start 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd052800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd04c102e20 0x7fd04c115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd052800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 0x7fd04c1162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd052800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1168f0 con 0x7fd04c1037c0 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd052800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c10fd00 con 0x7fd04c102e20 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd04aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 0x7fd04c1162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd04aa00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 0x7fd04c1162a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53472/0 (socket says 172.21.15.44:53472) 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.403+0000 7fd04aa00700 1 -- 172.21.15.44:0/117140135 learned_addr learned my addr 172.21.15.44:0/117140135 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:54:13.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd04b400700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd04c102e20 0x7fd04c115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd04aa00700 1 -- 172.21.15.44:0/117140135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd04c102e20 msgr2=0x7fd04c115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd04aa00700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd04c102e20 0x7fd04c115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd04aa00700 1 -- 172.21.15.44:0/117140135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c005040 con 0x7fd04c1037c0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd04aa00700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 0x7fd04c1162a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c800 tx=0x7fd038004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd03800a760 con 0x7fd04c1037c0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd03800ad90 con 0x7fd04c1037c0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0380113a0 con 0x7fd04c1037c0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.404+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c10ffe0 con 0x7fd04c1037c0 2024-04-16T11:54:13.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.405+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c1105a0 con 0x7fd04c1037c0 2024-04-16T11:54:13.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.406+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c107320 con 0x7fd04c1037c0 2024-04-16T11:54:13.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.407+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd038012400 con 0x7fd04c1037c0 2024-04-16T11:54:13.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.408+0000 7fd043400700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03006ec10 0x7fd0300710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:13.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.408+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fd03808eae0 con 0x7fd04c1037c0 2024-04-16T11:54:13.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.408+0000 7fd04b400700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03006ec10 0x7fd0300710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:13.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.409+0000 7fd04b400700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03006ec10 0x7fd0300710c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00b020 tx=0x7fd03c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:13.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.414+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd038057690 con 0x7fd04c1037c0 2024-04-16T11:54:13.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.667+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd04c1113f0 con 0x7fd03006ec10 2024-04-16T11:54:13.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.670+0000 7fd043400700 1 -- 172.21.15.44:0/117140135 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fd04c1113f0 con 0x7fd03006ec10 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": null, 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": false, 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "", 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": null, 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-04-16T11:54:13.667 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:54:13.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03006ec10 msgr2=0x7fd0300710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:13.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03006ec10 0x7fd0300710c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00b020 tx=0x7fd03c009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:13.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 msgr2=0x7fd04c1162a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:13.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 0x7fd04c1162a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c800 tx=0x7fd038004470 comp rx=0 tx=0).stop 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 shutdown_connections 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03006ec10 0x7fd0300710c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd04c102e20 0x7fd04c115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 --2- 172.21.15.44:0/117140135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd04c1037c0 0x7fd04c1162a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 >> 172.21.15.44:0/117140135 conn(0x7fd04c0f89d0 msgr2=0x7fd04c070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.673+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 shutdown_connections 2024-04-16T11:54:13.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:13.674+0000 7fd052800700 1 -- 172.21.15.44:0/117140135 wait complete. 2024-04-16T11:54:14.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:14 smithi045 ceph-mon[81423]: pgmap v332: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:14.404 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:14 smithi044 ceph-mon[93020]: pgmap v332: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:14.481 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph health detail' 2024-04-16T11:54:15.008 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:54:15.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:15 smithi045 ceph-mon[81423]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:54:15.338 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:15 smithi044 ceph-mon[93020]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:54:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:16 smithi045 ceph-mon[81423]: pgmap v333: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:16.414 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:16 smithi044 ceph-mon[93020]: pgmap v333: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 -- 172.21.15.44:0/3850950164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070c10 msgr2=0x7ff470108fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 --2- 172.21.15.44:0/3850950164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070c10 0x7ff470108fd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff45c004970 tx=0x7ff45c009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:17.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 -- 172.21.15.44:0/3850950164 shutdown_connections 2024-04-16T11:54:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 --2- 172.21.15.44:0/3850950164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070c10 0x7ff470108fd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 --2- 172.21.15.44:0/3850950164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070300 0x7ff4700706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 -- 172.21.15.44:0/3850950164 >> 172.21.15.44:0/3850950164 conn(0x7ff4700f87d0 msgr2=0x7ff4700fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 -- 172.21.15.44:0/3850950164 shutdown_connections 2024-04-16T11:54:17.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.291+0000 7ff475400700 1 -- 172.21.15.44:0/3850950164 wait complete. 2024-04-16T11:54:17.292 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:17 smithi044 ceph-mon[93020]: pgmap v334: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.292+0000 7ff475400700 1 Processor -- start 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.292+0000 7ff475400700 1 -- start start 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff475400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 0x7ff470115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff475400700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070c10 0x7ff4701160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff475400700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff470116710 con 0x7ff470070300 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff475400700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47010fb20 con 0x7ff470070c10 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff46fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 0x7ff470115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff46fe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 0x7ff470115b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53500/0 (socket says 172.21.15.44:53500) 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.293+0000 7ff46fe00700 1 -- 172.21.15.44:0/2441686324 learned_addr learned my addr 172.21.15.44:0/2441686324 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:54:17.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.294+0000 7ff46f400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070c10 0x7ff4701160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.294+0000 7ff46fe00700 1 -- 172.21.15.44:0/2441686324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070c10 msgr2=0x7ff4701160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.294+0000 7ff46fe00700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070c10 0x7ff4701160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.294+0000 7ff46fe00700 1 -- 172.21.15.44:0/2441686324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c005040 con 0x7ff470070300 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.294+0000 7ff46f400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070c10 0x7ff4701160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.294+0000 7ff46fe00700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 0x7ff470115b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff460002a50 tx=0x7ff4600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.295+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff460007cc0 con 0x7ff470070300 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.295+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff46000f040 con 0x7ff470070300 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.295+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47010fe00 con 0x7ff470070300 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.295+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff460009640 con 0x7ff470070300 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.295+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff470110310 con 0x7ff470070300 2024-04-16T11:54:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.296+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff470071a40 con 0x7ff470070300 2024-04-16T11:54:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.298+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff4600097a0 con 0x7ff470070300 2024-04-16T11:54:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.299+0000 7ff467e00700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff45406ec00 0x7ff4540710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.299+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7ff46008ee70 con 0x7ff470070300 2024-04-16T11:54:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.300+0000 7ff46f400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff45406ec00 0x7ff4540710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.300+0000 7ff46f400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff45406ec00 0x7ff4540710b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff45c004f50 tx=0x7ff45c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.304+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff460057a20 con 0x7ff470070300 2024-04-16T11:54:17.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:17 smithi045 ceph-mon[81423]: pgmap v334: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.632+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff47004bf30 con 0x7ff470070300 2024-04-16T11:54:17.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.634+0000 7ff467e00700 1 -- 172.21.15.44:0/2441686324 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff46005b0e0 con 0x7ff470070300 2024-04-16T11:54:17.633 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-04-16T11:54:17.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.636+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff45406ec00 msgr2=0x7ff4540710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:17.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.636+0000 7ff475400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff45406ec00 0x7ff4540710b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff45c004f50 tx=0x7ff45c009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:17.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.636+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 msgr2=0x7ff470115b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.636+0000 7ff475400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 0x7ff470115b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff460002a50 tx=0x7ff4600046b0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 shutdown_connections 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff45406ec00 0x7ff4540710b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff470070300 0x7ff470115b80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 --2- 172.21.15.44:0/2441686324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff470070c10 0x7ff4701160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 >> 172.21.15.44:0/2441686324 conn(0x7ff4700f87d0 msgr2=0x7ff4700f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 shutdown_connections 2024-04-16T11:54:17.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:17.637+0000 7ff475400700 1 -- 172.21.15.44:0/2441686324 wait complete. 2024-04-16T11:54:18.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:18 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2441686324' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:54:18.370 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:18 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2441686324' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T11:54:18.423 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions' 2024-04-16T11:54:18.920 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:54:19.304 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:19 smithi044 ceph-mon[93020]: pgmap v335: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:19.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:19 smithi045 ceph-mon[81423]: pgmap v335: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:21.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 -- 172.21.15.44:0/3537924377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9770105030 msgr2=0x7f9770105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:21.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 --2- 172.21.15.44:0/3537924377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9770105030 0x7f9770105400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9760004940 tx=0x7f9760009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 -- 172.21.15.44:0/3537924377 shutdown_connections 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 --2- 172.21.15.44:0/3537924377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97701059d0 0x7f97700ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 --2- 172.21.15.44:0/3537924377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9770105030 0x7f9770105400 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 -- 172.21.15.44:0/3537924377 >> 172.21.15.44:0/3537924377 conn(0x7f97700fabd0 msgr2=0x7f97700fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 -- 172.21.15.44:0/3537924377 shutdown_connections 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.164+0000 7f9776800700 1 -- 172.21.15.44:0/3537924377 wait complete. 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.165+0000 7f9776800700 1 Processor -- start 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.165+0000 7f9776800700 1 -- start start 2024-04-16T11:54:21.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f9776800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9770105030 0x7f9770115bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f9776800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 0x7f9770116100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f9776800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97701167e0 con 0x7f97701059d0 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f9776800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977010fd30 con 0x7f9770105030 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f976ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 0x7f9770116100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f976ea00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 0x7f9770116100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53510/0 (socket says 172.21.15.44:53510) 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f976ea00700 1 -- 172.21.15.44:0/2590450882 learned_addr learned my addr 172.21.15.44:0/2590450882 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.166+0000 7f976f400700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9770105030 0x7f9770115bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f976ea00700 1 -- 172.21.15.44:0/2590450882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9770105030 msgr2=0x7f9770115bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f976ea00700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9770105030 0x7f9770115bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f976ea00700 1 -- 172.21.15.44:0/2590450882 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760005040 con 0x7f97701059d0 2024-04-16T11:54:21.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f976ea00700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 0x7f9770116100 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f975c00b790 tx=0x7f975c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:21.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975c00f6f0 con 0x7f97701059d0 2024-04-16T11:54:21.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f975c00fd20 con 0x7f97701059d0 2024-04-16T11:54:21.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977010ffb0 con 0x7f97701059d0 2024-04-16T11:54:21.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975c0103a0 con 0x7f97701059d0 2024-04-16T11:54:21.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.167+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9770110570 con 0x7f97701059d0 2024-04-16T11:54:21.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.168+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9770100780 con 0x7f97701059d0 2024-04-16T11:54:21.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.170+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f975c011400 con 0x7f97701059d0 2024-04-16T11:54:21.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.171+0000 7f9767400700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f975406ec00 0x7f97540710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:54:21.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.171+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f975c08ea40 con 0x7f97701059d0 2024-04-16T11:54:21.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.171+0000 7f976f400700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f975406ec00 0x7f97540710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:54:21.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.172+0000 7f976f400700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f975406ec00 0x7f97540710b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f976000b020 tx=0x7f9760009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:54:21.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.176+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f975c05b020 con 0x7f97701059d0 2024-04-16T11:54:21.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.505+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f977010c230 con 0x7f97701059d0 2024-04-16T11:54:21.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.506+0000 7f9767400700 1 -- 172.21.15.44:0/2590450882 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f975c05abb0 con 0x7f97701059d0 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:54:21.504 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:54:21.505 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f975406ec00 msgr2=0x7f97540710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f975406ec00 0x7f97540710b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f976000b020 tx=0x7f9760009e30 comp rx=0 tx=0).stop 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 msgr2=0x7f9770116100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 0x7f9770116100 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f975c00b790 tx=0x7f975c009220 comp rx=0 tx=0).stop 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 shutdown_connections 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f975406ec00 0x7f97540710b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:21.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9770105030 0x7f9770115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:21.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 --2- 172.21.15.44:0/2590450882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f97701059d0 0x7f9770116100 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:54:21.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 >> 172.21.15.44:0/2590450882 conn(0x7f97700fabd0 msgr2=0x7f97700fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:54:21.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.509+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 shutdown_connections 2024-04-16T11:54:21.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:54:21.510+0000 7f9776800700 1 -- 172.21.15.44:0/2590450882 wait complete. 2024-04-16T11:54:22.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:21 smithi045 ceph-mon[81423]: pgmap v336: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:22.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:21 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2590450882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:54:22.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:21 smithi044 ceph-mon[93020]: pgmap v336: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:22.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:21 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2590450882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:54:22.372 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-16T11:54:22.882 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:54:24.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:23 smithi045 ceph-mon[81423]: pgmap v337: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:24.096 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:23 smithi044 ceph-mon[93020]: pgmap v337: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:25.103 INFO:teuthology.orchestra.run.smithi044.stdout:wait for servicemap items w/ changing names to refresh 2024-04-16T11:54:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:25 smithi044 ceph-mon[93020]: pgmap v338: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:25.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:26.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:25 smithi045 ceph-mon[81423]: pgmap v338: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:26.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:26.109 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'sleep 60' 2024-04-16T11:54:26.632 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:54:27.976 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:27 smithi044 ceph-mon[93020]: pgmap v339: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:28.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:27 smithi045 ceph-mon[81423]: pgmap v339: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:30.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:29 smithi045 ceph-mon[81423]: pgmap v340: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:30.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:29 smithi044 ceph-mon[93020]: pgmap v340: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:31.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:31 smithi045 ceph-mon[81423]: pgmap v341: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:31.725 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:31 smithi044 ceph-mon[93020]: pgmap v341: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:34.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:33 smithi045 ceph-mon[81423]: pgmap v342: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:34.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:33 smithi044 ceph-mon[93020]: pgmap v342: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:36.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:35 smithi045 ceph-mon[81423]: pgmap v343: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:36.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:35 smithi044 ceph-mon[93020]: pgmap v343: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:37.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:36 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:54:37.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:36 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:54:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:37 smithi045 ceph-mon[81423]: pgmap v344: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:37 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:54:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:37 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:54:38.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:37 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:54:38.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:37 smithi044 ceph-mon[93020]: pgmap v344: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:37 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:54:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:37 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:54:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:37 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:54:40.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:39 smithi045 ceph-mon[81423]: pgmap v345: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:40.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:39 smithi044 ceph-mon[93020]: pgmap v345: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:41.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:41.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:42.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:41 smithi045 ceph-mon[81423]: pgmap v346: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:42.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:41 smithi044 ceph-mon[93020]: pgmap v346: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:44.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:43 smithi045 ceph-mon[81423]: pgmap v347: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:44.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:43 smithi044 ceph-mon[93020]: pgmap v347: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:45.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:45 smithi044 ceph-mon[93020]: pgmap v348: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:46.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:45 smithi045 ceph-mon[81423]: pgmap v348: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:47 smithi044 ceph-mon[93020]: pgmap v349: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:47 smithi045 ceph-mon[81423]: pgmap v349: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:50.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:49 smithi044 ceph-mon[93020]: pgmap v350: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:50.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:49 smithi045 ceph-mon[81423]: pgmap v350: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:52.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:51 smithi044 ceph-mon[93020]: pgmap v351: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:51 smithi045 ceph-mon[81423]: pgmap v351: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:53 smithi044 ceph-mon[93020]: pgmap v352: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:54.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:53 smithi045 ceph-mon[81423]: pgmap v352: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:55 smithi044 ceph-mon[93020]: pgmap v353: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:55 smithi045 ceph-mon[81423]: pgmap v353: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:54:58.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:57 smithi044 ceph-mon[93020]: pgmap v354: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:54:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:57 smithi045 ceph-mon[81423]: pgmap v354: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:54:59 smithi044 ceph-mon[93020]: pgmap v355: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:54:59 smithi045 ceph-mon[81423]: pgmap v355: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:01 smithi044 ceph-mon[93020]: pgmap v356: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:01 smithi045 ceph-mon[81423]: pgmap v356: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:03 smithi044 ceph-mon[93020]: pgmap v357: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:03 smithi045 ceph-mon[81423]: pgmap v357: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:06.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:05 smithi044 ceph-mon[93020]: pgmap v358: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:05 smithi045 ceph-mon[81423]: pgmap v358: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:07 smithi044 ceph-mon[93020]: pgmap v359: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:07 smithi045 ceph-mon[81423]: pgmap v359: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:10.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:09 smithi044 ceph-mon[93020]: pgmap v360: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:10.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:09 smithi045 ceph-mon[81423]: pgmap v360: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:11.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:12.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:11 smithi044 ceph-mon[93020]: pgmap v361: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:12.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:11 smithi045 ceph-mon[81423]: pgmap v361: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:14.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:13 smithi044 ceph-mon[93020]: pgmap v362: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:14.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:13 smithi045 ceph-mon[81423]: pgmap v362: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:15 smithi044 ceph-mon[93020]: pgmap v363: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:15 smithi045 ceph-mon[81423]: pgmap v363: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:17 smithi044 ceph-mon[93020]: pgmap v364: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:18.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:17 smithi045 ceph-mon[81423]: pgmap v364: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:20.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:19 smithi044 ceph-mon[93020]: pgmap v365: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:20.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:19 smithi045 ceph-mon[81423]: pgmap v365: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:22.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:21 smithi044 ceph-mon[93020]: pgmap v366: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:22.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:21 smithi045 ceph-mon[81423]: pgmap v366: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:24.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:23 smithi044 ceph-mon[93020]: pgmap v367: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:24.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:23 smithi045 ceph-mon[81423]: pgmap v367: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:26.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:25 smithi044 ceph-mon[93020]: pgmap v368: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:26.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:25 smithi045 ceph-mon[81423]: pgmap v368: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:28.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:27 smithi044 ceph-mon[93020]: pgmap v369: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:28.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:27 smithi045 ceph-mon[81423]: pgmap v369: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:29.610 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch ps' 2024-04-16T11:55:30.142 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:55:30.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:29 smithi044 ceph-mon[93020]: pgmap v370: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:30.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:29 smithi045 ceph-mon[81423]: pgmap v370: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:32.237 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:31 smithi044 ceph-mon[93020]: pgmap v371: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:32.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:31 smithi045 ceph-mon[81423]: pgmap v371: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:32.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.541+0000 7f74c7200700 1 -- 172.21.15.44:0/3136111749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c01059d0 msgr2=0x7f74c00ff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.541+0000 7f74c7200700 1 --2- 172.21.15.44:0/3136111749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c01059d0 0x7f74c00ff090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f74ac004940 tx=0x7f74ac009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.542+0000 7f74c7200700 1 -- 172.21.15.44:0/3136111749 shutdown_connections 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.542+0000 7f74c7200700 1 --2- 172.21.15.44:0/3136111749 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c01059d0 0x7f74c00ff090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.542+0000 7f74c7200700 1 --2- 172.21.15.44:0/3136111749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74c0105030 0x7f74c0105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.542+0000 7f74c7200700 1 -- 172.21.15.44:0/3136111749 >> 172.21.15.44:0/3136111749 conn(0x7f74c00fabd0 msgr2=0x7f74c00fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.542+0000 7f74c7200700 1 -- 172.21.15.44:0/3136111749 shutdown_connections 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.542+0000 7f74c7200700 1 -- 172.21.15.44:0/3136111749 wait complete. 2024-04-16T11:55:32.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.543+0000 7f74c7200700 1 Processor -- start 2024-04-16T11:55:32.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.543+0000 7f74c7200700 1 -- start start 2024-04-16T11:55:32.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74c7200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 0x7f74c0115b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:32.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74c7200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74c01059d0 0x7f74c01160d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:32.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74c7200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0116720 con 0x7f74c0105030 2024-04-16T11:55:32.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74c7200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c010fd00 con 0x7f74c01059d0 2024-04-16T11:55:32.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74bfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 0x7f74c0115b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:32.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74bfe00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 0x7f74c0115b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41640/0 (socket says 172.21.15.44:41640) 2024-04-16T11:55:32.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74bfe00700 1 -- 172.21.15.44:0/2167116468 learned_addr learned my addr 172.21.15.44:0/2167116468 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.544+0000 7f74bf400700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74c01059d0 0x7f74c01160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74bfe00700 1 -- 172.21.15.44:0/2167116468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74c01059d0 msgr2=0x7f74c01160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74bfe00700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74c01059d0 0x7f74c01160d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74bfe00700 1 -- 172.21.15.44:0/2167116468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac005040 con 0x7f74c0105030 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74bfe00700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 0x7f74c0115b90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c830 tx=0x7f74b0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b000a760 con 0x7f74c0105030 2024-04-16T11:55:32.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74b000ad90 con 0x7f74c0105030 2024-04-16T11:55:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b00113a0 con 0x7f74c0105030 2024-04-16T11:55:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c010ffe0 con 0x7f74c0105030 2024-04-16T11:55:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.545+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c01105a0 con 0x7f74c0105030 2024-04-16T11:55:32.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.546+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c0100780 con 0x7f74c0105030 2024-04-16T11:55:32.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.548+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74b0012400 con 0x7f74c0105030 2024-04-16T11:55:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.548+0000 7f74b7e00700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f74a406ec00 0x7f74a40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.549+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f74b008eb10 con 0x7f74c0105030 2024-04-16T11:55:32.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.549+0000 7f74bf400700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f74a406ec00 0x7f74a40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:32.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.550+0000 7f74bf400700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f74a406ec00 0x7f74a40710b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00b020 tx=0x7f74ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:32.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.554+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f74b005b1f0 con 0x7f74c0105030 2024-04-16T11:55:32.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.796+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 --> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c0111250 con 0x7f74a406ec00 2024-04-16T11:55:32.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.827+0000 7f74b7e00700 1 -- 172.21.15.44:0/2167116468 <== mgr.34104 v2:172.21.15.44:6800/2084563884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f74c0111250 con 0x7f74a406ec00 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.smithi044 smithi044 *:9093,9094 running (2m) 2m ago 23m 14.2M - 0.25.0 c8568f914cd2 42392ee6c260 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi044 smithi044 running (9m) 2m ago 23m 6660k - 19.0.0-2884-g218a6914 a4e4811815e4 1660597e3208 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:crash.smithi045 smithi045 running (9m) 3m ago 21m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 6cc7b971153e 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.smithi044 smithi044 *:3000 running (2m) 2m ago 22m 50.6M - 9.4.12 2bacad6d85d8 29f522faff8a 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.hlafhv smithi044 running (5m) 2m ago 15m 12.1M - 19.0.0-2884-g218a6914 a4e4811815e4 7e42996643b3 2024-04-16T11:55:32.825 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi044.ufhhlc smithi044 running (5m) 2m ago 16m 21.8M - 19.0.0-2884-g218a6914 a4e4811815e4 426ee20d4be1 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.hxlkks smithi045 running (4m) 3m ago 15m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 e08aedf9cf6c 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:mds.cephfs.smithi045.noofsd smithi045 running (4m) 3m ago 15m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 3d9075593f9c 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi044.sczobv smithi044 *:8443,9283,8765 running (12m) 2m ago 24m 520M - 19.0.0-2884-g218a6914 a4e4811815e4 c072b460f21f 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.smithi045.sdlwif smithi045 *:8443,9283,8765 running (11m) 3m ago 21m 455M - 19.0.0-2884-g218a6914 a4e4811815e4 0cef9288f9d1 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi044 smithi044 running (11m) 2m ago 25m 56.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5d963afffb61 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:mon.smithi045 smithi045 running (10m) 3m ago 21m 39.8M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 72db4f2396ce 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi044 smithi044 *:9100 running (3m) 2m ago 23m 7377k - 1.5.0 0da6a335fe13 d6c7c8a1653a 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.smithi045 smithi045 *:9100 running (3m) 3m ago 21m 3007k - 1.5.0 0da6a335fe13 e9596e00b617 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (8m) 2m ago 20m 55.3M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93ed878736d9 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (8m) 2m ago 20m 46.2M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 93c9db3f124e 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (7m) 2m ago 19m 45.6M 1559M 19.0.0-2884-g218a6914 a4e4811815e4 b3bcfc1b4f97 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi045 running (7m) 3m ago 18m 45.2M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 be3375e68a6a 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi045 running (6m) 3m ago 18m 42.6M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 c6a09eb63df3 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi045 running (6m) 3m ago 17m 49.7M 2583M 19.0.0-2884-g218a6914 a4e4811815e4 114e11d6d49a 2024-04-16T11:55:32.826 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.smithi044 smithi044 *:9095 running (3m) 2m ago 22m 48.6M - 2.43.0 a07b618ecd1d b3776ca300db 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f74a406ec00 msgr2=0x7f74a40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f74a406ec00 0x7f74a40710b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00b020 tx=0x7f74ac009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 msgr2=0x7f74c0115b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 0x7f74c0115b90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c830 tx=0x7f74b0004470 comp rx=0 tx=0).stop 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 shutdown_connections 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f74a406ec00 0x7f74a40710b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:32.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74c0105030 0x7f74c0115b90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:32.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 --2- 172.21.15.44:0/2167116468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74c01059d0 0x7f74c01160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:32.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 >> 172.21.15.44:0/2167116468 conn(0x7f74c00fabd0 msgr2=0x7f74c00fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:32.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 shutdown_connections 2024-04-16T11:55:32.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:32.831+0000 7f74c7200700 1 -- 172.21.15.44:0/2167116468 wait complete. 2024-04-16T11:55:33.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions' 2024-04-16T11:55:34.120 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:55:34.266 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:33 smithi044 ceph-mon[93020]: pgmap v372: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:34.266 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:33 smithi044 ceph-mon[93020]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:55:34.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:33 smithi045 ceph-mon[81423]: pgmap v372: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:34.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:33 smithi045 ceph-mon[81423]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T11:55:36.108 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:35 smithi044 ceph-mon[93020]: pgmap v373: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.283+0000 7f7dca800700 1 -- 172.21.15.44:0/2231274379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 msgr2=0x7f7dc40fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.283+0000 7f7dca800700 1 --2- 172.21.15.44:0/2231274379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc40fcc50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7db0004940 tx=0x7f7db0009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.284+0000 7f7dca800700 1 -- 172.21.15.44:0/2231274379 shutdown_connections 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.284+0000 7f7dca800700 1 --2- 172.21.15.44:0/2231274379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc40fcc50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.284+0000 7f7dca800700 1 --2- 172.21.15.44:0/2231274379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dc4102c40 0x7f7dc4103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.284+0000 7f7dca800700 1 -- 172.21.15.44:0/2231274379 >> 172.21.15.44:0/2231274379 conn(0x7f7dc40f8790 msgr2=0x7f7dc40faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.284+0000 7f7dca800700 1 -- 172.21.15.44:0/2231274379 shutdown_connections 2024-04-16T11:55:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.284+0000 7f7dca800700 1 -- 172.21.15.44:0/2231274379 wait complete. 2024-04-16T11:55:36.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.285+0000 7f7dca800700 1 Processor -- start 2024-04-16T11:55:36.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.285+0000 7f7dca800700 1 -- start start 2024-04-16T11:55:36.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.285+0000 7f7dca800700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dc4102c40 0x7f7dc40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:36.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.285+0000 7f7dca800700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:36.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.285+0000 7f7dca800700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4070e20 con 0x7f7dc41035e0 2024-04-16T11:55:36.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.285+0000 7f7dca800700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4070f60 con 0x7f7dc4102c40 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc8a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc8a00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41654/0 (socket says 172.21.15.44:41654) 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc8a00700 1 -- 172.21.15.44:0/1537035335 learned_addr learned my addr 172.21.15.44:0/1537035335 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc9400700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dc4102c40 0x7f7dc40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc8a00700 1 -- 172.21.15.44:0/1537035335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dc4102c40 msgr2=0x7f7dc40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc8a00700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dc4102c40 0x7f7dc40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.286+0000 7f7dc8a00700 1 -- 172.21.15.44:0/1537035335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db0005040 con 0x7f7dc41035e0 2024-04-16T11:55:36.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.287+0000 7f7dc8a00700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc4070800 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009100 tx=0x7f7db0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:36.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.287+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7db000e040 con 0x7f7dc41035e0 2024-04-16T11:55:36.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.287+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7db0013420 con 0x7f7dc41035e0 2024-04-16T11:55:36.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.287+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7db001c630 con 0x7f7dc41035e0 2024-04-16T11:55:36.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.287+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc4119ce0 con 0x7f7dc41035e0 2024-04-16T11:55:36.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.287+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc411a1c0 con 0x7f7dc41035e0 2024-04-16T11:55:36.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.288+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc40fe390 con 0x7f7dc41035e0 2024-04-16T11:55:36.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.289+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7db0013590 con 0x7f7dc41035e0 2024-04-16T11:55:36.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.290+0000 7f7dc1600700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7dac06ec00 0x7f7dac0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:36.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.290+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f7db008fdc0 con 0x7f7dc41035e0 2024-04-16T11:55:36.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.291+0000 7f7dc9400700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7dac06ec00 0x7f7dac0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:36.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.292+0000 7f7dc9400700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7dac06ec00 0x7f7dac0710b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7db8006fd0 tx=0x7f7db8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:36.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.296+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7db0093030 con 0x7f7dc41035e0 2024-04-16T11:55:36.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:35 smithi045 ceph-mon[81423]: pgmap v373: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:36.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.628+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7dc404bed0 con 0x7f7dc41035e0 2024-04-16T11:55:36.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.630+0000 7f7dc1600700 1 -- 172.21.15.44:0/1537035335 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7db00588f0 con 0x7f7dc41035e0 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": { 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-04-16T11:55:36.628 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T11:55:36.629 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-16T11:55:36.629 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.632+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7dac06ec00 msgr2=0x7f7dac0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.632+0000 7f7dca800700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7dac06ec00 0x7f7dac0710b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7db8006fd0 tx=0x7f7db8008040 comp rx=0 tx=0).stop 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 msgr2=0x7f7dc4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc4070800 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009100 tx=0x7f7db0009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 shutdown_connections 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f7dac06ec00 0x7f7dac0710b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dc4102c40 0x7f7dc40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 --2- 172.21.15.44:0/1537035335 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dc41035e0 0x7f7dc4070800 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 >> 172.21.15.44:0/1537035335 conn(0x7f7dc40f8790 msgr2=0x7f7dc40f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 shutdown_connections 2024-04-16T11:55:36.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:36.633+0000 7f7dca800700 1 -- 172.21.15.44:0/1537035335 wait complete. 2024-04-16T11:55:37.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:37 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1537035335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:55:37.399 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:37 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1537035335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:55:37.469 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-16T11:55:37.983 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:55:38.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:38 smithi045 ceph-mon[81423]: pgmap v374: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:38.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:38 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:55:38.361 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:38 smithi044 ceph-mon[93020]: pgmap v374: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:38.361 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:38 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.179+0000 7f7180600700 1 -- 172.21.15.44:0/3855512200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7178102e60 msgr2=0x7f7178103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.179+0000 7f7180600700 1 --2- 172.21.15.44:0/3855512200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7178102e60 0x7f7178103230 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f716c004940 tx=0x7f716c009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.180+0000 7f7180600700 1 -- 172.21.15.44:0/3855512200 shutdown_connections 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.180+0000 7f7180600700 1 --2- 172.21.15.44:0/3855512200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71780656b0 0x7f7178065b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.180+0000 7f7180600700 1 --2- 172.21.15.44:0/3855512200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7178102e60 0x7f7178103230 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.180+0000 7f7180600700 1 -- 172.21.15.44:0/3855512200 >> 172.21.15.44:0/3855512200 conn(0x7f7178072bc0 msgr2=0x7f7178072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:40.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.180+0000 7f7180600700 1 -- 172.21.15.44:0/3855512200 shutdown_connections 2024-04-16T11:55:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.180+0000 7f7180600700 1 -- 172.21.15.44:0/3855512200 wait complete. 2024-04-16T11:55:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.181+0000 7f7180600700 1 Processor -- start 2024-04-16T11:55:40.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.181+0000 7f7180600700 1 -- start start 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.181+0000 7f7180600700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 0x7f7178115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f7180600700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7178102e60 0x7f71781162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f7180600700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71781168f0 con 0x7f71780656b0 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f7180600700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717810fcc0 con 0x7f7178102e60 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 0x7f7178115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717dc00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 0x7f7178115d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41664/0 (socket says 172.21.15.44:41664) 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717dc00700 1 -- 172.21.15.44:0/1806174243 learned_addr learned my addr 172.21.15.44:0/1806174243 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717d200700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7178102e60 0x7f71781162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:40.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717dc00700 1 -- 172.21.15.44:0/1806174243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7178102e60 msgr2=0x7f71781162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717dc00700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7178102e60 0x7f71781162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717dc00700 1 -- 172.21.15.44:0/1806174243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c005040 con 0x7f71780656b0 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.182+0000 7f717d200700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7178102e60 0x7f71781162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.183+0000 7f717dc00700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 0x7f7178115d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f716c009150 tx=0x7f716c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.183+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f716c00e040 con 0x7f71780656b0 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.183+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f716c013420 con 0x7f71780656b0 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.183+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717810ff40 con 0x7f71780656b0 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.183+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f716c01c630 con 0x7f71780656b0 2024-04-16T11:55:40.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.183+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7178110500 con 0x7f71780656b0 2024-04-16T11:55:40.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.185+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178106a70 con 0x7f71780656b0 2024-04-16T11:55:40.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.186+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f716c013590 con 0x7f71780656b0 2024-04-16T11:55:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.187+0000 7f7175600700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f716006ec10 0x7f71600710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.188+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f716c08fd50 con 0x7f71780656b0 2024-04-16T11:55:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.188+0000 7f717d200700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f716006ec10 0x7f71600710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:40.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.189+0000 7f717d200700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f716006ec10 0x7f71600710c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7164009fd0 tx=0x7f716400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:40.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.193+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f716c05c430 con 0x7f71780656b0 2024-04-16T11:55:40.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:40 smithi045 ceph-mon[81423]: pgmap v375: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:55:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:55:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:55:40.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:40 smithi044 ceph-mon[93020]: pgmap v375: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:40.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:55:40.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:55:40.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:55:40.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.538+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7178106a70 con 0x7f71780656b0 2024-04-16T11:55:40.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.539+0000 7f7175600700 1 -- 172.21.15.44:0/1806174243 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f716c05bfc0 con 0x7f71780656b0 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f716006ec10 msgr2=0x7f71600710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f716006ec10 0x7f71600710c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7164009fd0 tx=0x7f716400b040 comp rx=0 tx=0).stop 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 msgr2=0x7f7178115d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 0x7f7178115d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f716c009150 tx=0x7f716c009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 shutdown_connections 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f716006ec10 0x7f71600710c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f71780656b0 0x7f7178115d60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:40.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 --2- 172.21.15.44:0/1806174243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7178102e60 0x7f71781162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:40.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.542+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 >> 172.21.15.44:0/1806174243 conn(0x7f7178072bc0 msgr2=0x7f71780fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:40.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.543+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 shutdown_connections 2024-04-16T11:55:40.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:40.543+0000 7f7180600700 1 -- 172.21.15.44:0/1806174243 wait complete. 2024-04-16T11:55:40.557 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-04-16T11:55:41.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:41.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:41 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1806174243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:55:41.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:41.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:41 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1806174243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:55:41.413 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-16T11:55:41.941 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:55:42.309 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:42 smithi044 ceph-mon[93020]: pgmap v376: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:42 smithi045 ceph-mon[81423]: pgmap v376: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:44.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.239+0000 7f32f2200700 1 -- 172.21.15.44:0/3685330880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 msgr2=0x7f32ec0fbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:44.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.239+0000 7f32f2200700 1 --2- 172.21.15.44:0/3685330880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec0fbf40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f32dc004940 tx=0x7f32dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:44.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.240+0000 7f32f2200700 1 -- 172.21.15.44:0/3685330880 shutdown_connections 2024-04-16T11:55:44.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.240+0000 7f32f2200700 1 --2- 172.21.15.44:0/3685330880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 0x7f32ec1091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:44.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.240+0000 7f32f2200700 1 --2- 172.21.15.44:0/3685330880 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec0fbf40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.240+0000 7f32f2200700 1 -- 172.21.15.44:0/3685330880 >> 172.21.15.44:0/3685330880 conn(0x7f32ec0f78a0 msgr2=0x7f32ec0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.240+0000 7f32f2200700 1 -- 172.21.15.44:0/3685330880 shutdown_connections 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.240+0000 7f32f2200700 1 -- 172.21.15.44:0/3685330880 wait complete. 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.241+0000 7f32f2200700 1 Processor -- start 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.241+0000 7f32f2200700 1 -- start start 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.241+0000 7f32f2200700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.241+0000 7f32f2200700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 0x7f32ec1162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.241+0000 7f32f2200700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec116900 con 0x7f32ec0fbb70 2024-04-16T11:55:44.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.241+0000 7f32f2200700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec10fcd0 con 0x7f32ec0fc480 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec115d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58068/0 (socket says 172.21.15.44:58068) 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 -- 172.21.15.44:0/2003307529 learned_addr learned my addr 172.21.15.44:0/2003307529 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eaa00700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 0x7f32ec1162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 -- 172.21.15.44:0/2003307529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 msgr2=0x7f32ec1162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 0x7f32ec1162b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:44.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 -- 172.21.15.44:0/2003307529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc005040 con 0x7f32ec0fbb70 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eaa00700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 0x7f32ec1162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.242+0000 7f32eb400700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec115d70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00b8b0 tx=0x7f32dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.243+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32dc00e040 con 0x7f32ec0fbb70 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.243+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ec10ff50 con 0x7f32ec0fbb70 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.243+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32dc004290 con 0x7f32ec0fbb70 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.243+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32dc01c3f0 con 0x7f32ec0fbb70 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.243+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec110510 con 0x7f32ec0fbb70 2024-04-16T11:55:44.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.244+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec066780 con 0x7f32ec0fbb70 2024-04-16T11:55:44.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.245+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32dc0137b0 con 0x7f32ec0fbb70 2024-04-16T11:55:44.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.246+0000 7f32e3400700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32d006ec00 0x7f32d00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:44.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.246+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f32dc08fed0 con 0x7f32ec0fbb70 2024-04-16T11:55:44.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.247+0000 7f32eaa00700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32d006ec00 0x7f32d00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:44.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.247+0000 7f32eaa00700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32d006ec00 0x7f32d00710b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f32d8007fd0 tx=0x7f32d8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:44.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.256+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f32dc05c530 con 0x7f32ec0fbb70 2024-04-16T11:55:44.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:44 smithi045 ceph-mon[81423]: pgmap v377: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:44.426 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:44 smithi044 ceph-mon[93020]: pgmap v377: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:44.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.589+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f32ec04bed0 con 0x7f32ec0fbb70 2024-04-16T11:55:44.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.591+0000 7f32e3400700 1 -- 172.21.15.44:0/2003307529 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f32dc05c0c0 con 0x7f32ec0fbb70 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.593+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32d006ec00 msgr2=0x7f32d00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.593+0000 7f32f2200700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32d006ec00 0x7f32d00710b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f32d8007fd0 tx=0x7f32d8009040 comp rx=0 tx=0).stop 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.593+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 msgr2=0x7f32ec115d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.593+0000 7f32f2200700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec115d70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00b8b0 tx=0x7f32dc009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 shutdown_connections 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32d006ec00 0x7f32d00710b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32ec0fbb70 0x7f32ec115d70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 --2- 172.21.15.44:0/2003307529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32ec0fc480 0x7f32ec1162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 >> 172.21.15.44:0/2003307529 conn(0x7f32ec0f78a0 msgr2=0x7f32ec0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 shutdown_connections 2024-04-16T11:55:44.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:44.594+0000 7f32f2200700 1 -- 172.21.15.44:0/2003307529 wait complete. 2024-04-16T11:55:44.608 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)" 2024-04-16T11:55:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:45 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2003307529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:55:45.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:45 smithi045 ceph-mon[81423]: pgmap v378: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:45.350 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:45 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2003307529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T11:55:45.350 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:45 smithi044 ceph-mon[93020]: pgmap v378: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:45.401 DEBUG:teuthology.parallel:result is None 2024-04-16T11:55:45.401 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T11:55:45.413 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T11:55:45.414 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T11:55:45.900 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/mon.smithi044/config 2024-04-16T11:55:48.055 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:47 smithi044 ceph-mon[93020]: pgmap v379: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:48.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:47 smithi045 ceph-mon[81423]: pgmap v379: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 -- 172.21.15.44:0/300179214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 msgr2=0x7f343c0fbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 --2- 172.21.15.44:0/300179214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c0fbe00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f342c004940 tx=0x7f342c009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 -- 172.21.15.44:0/300179214 shutdown_connections 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 --2- 172.21.15.44:0/300179214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 0x7f343c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 --2- 172.21.15.44:0/300179214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c0fbe00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 -- 172.21.15.44:0/300179214 >> 172.21.15.44:0/300179214 conn(0x7f343c0f76b0 msgr2=0x7f343c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 -- 172.21.15.44:0/300179214 shutdown_connections 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.366+0000 7f3440c00700 1 -- 172.21.15.44:0/300179214 wait complete. 2024-04-16T11:55:48.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.367+0000 7f3440c00700 1 Processor -- start 2024-04-16T11:55:48.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.367+0000 7f3440c00700 1 -- start start 2024-04-16T11:55:48.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f3440c00700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c119ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f3440c00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 0x7f343c11a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f3440c00700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c11ab80 con 0x7f343c0fba30 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f3440c00700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c113f50 con 0x7f343c0fc3d0 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f343b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c119ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f343b400700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c119ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58074/0 (socket says 172.21.15.44:58074) 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f343b400700 1 -- 172.21.15.44:0/1954798383 learned_addr learned my addr 172.21.15.44:0/1954798383 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.368+0000 7f343aa00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 0x7f343c11a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f343b400700 1 -- 172.21.15.44:0/1954798383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 msgr2=0x7f343c11a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f343b400700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 0x7f343c11a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f343b400700 1 -- 172.21.15.44:0/1954798383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c005040 con 0x7f343c0fba30 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f343aa00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 0x7f343c11a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:48.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f343b400700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c119ff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f342c009150 tx=0x7f342c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:48.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f342c00e040 con 0x7f343c0fba30 2024-04-16T11:55:48.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f342c013420 con 0x7f343c0fba30 2024-04-16T11:55:48.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c1141d0 con 0x7f343c0fba30 2024-04-16T11:55:48.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f342c01c630 con 0x7f343c0fba30 2024-04-16T11:55:48.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.369+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c114710 con 0x7f343c0fba30 2024-04-16T11:55:48.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.371+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c066780 con 0x7f343c0fba30 2024-04-16T11:55:48.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.372+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f342c013590 con 0x7f343c0fba30 2024-04-16T11:55:48.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.373+0000 7f3433400700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f342006ec00 0x7f34200710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:48.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.373+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f342c08fe30 con 0x7f343c0fba30 2024-04-16T11:55:48.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.373+0000 7f343aa00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f342006ec00 0x7f34200710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:48.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.374+0000 7f343aa00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f342006ec00 0x7f34200710b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3428007fd0 tx=0x7f3428009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:48.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.378+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f342c05c490 con 0x7f343c0fba30 2024-04-16T11:55:48.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.663+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f343c05f5f0 con 0x7f343c0fba30 2024-04-16T11:55:48.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.664+0000 7f3433400700 1 -- 172.21.15.44:0/1954798383 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 37 v37) v1 ==== 76+0+1932 (secure 0 0 0) 0x7f342c05c020 con 0x7f343c0fba30 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:e37 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 36 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-16T11:39:27.511557+0000 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-16T11:51:22.505053+0000 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-16T11:55:48.663 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 60 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 91 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=34432} 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3] 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-16T11:55:48.664 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 1 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 34432 members: 34432 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.ufhhlc{0:34432} state up:active seq 24 join_fscid=1 addr [v2:172.21.15.44:6826/1916629246,v1:172.21.15.44:6827/1916629246] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi044.hlafhv{-1:34464} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.44:6828/2913937513,v1:172.21.15.44:6829/2913937513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.noofsd{-1:44319} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6824/33255321,v1:172.21.15.45:6825/33255321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:55:48.665 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.cephfs.smithi045.hxlkks{-1:44341} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6826/3167580677,v1:172.21.15.45:6827/3167580677] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.667+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f342006ec00 msgr2=0x7f34200710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.667+0000 7f3440c00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f342006ec00 0x7f34200710b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3428007fd0 tx=0x7f3428009040 comp rx=0 tx=0).stop 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.667+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 msgr2=0x7f343c119ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.667+0000 7f3440c00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c119ff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f342c009150 tx=0x7f342c009e30 comp rx=0 tx=0).stop 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 shutdown_connections 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f342006ec00 0x7f34200710b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f343c0fba30 0x7f343c119ff0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 --2- 172.21.15.44:0/1954798383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f343c0fc3d0 0x7f343c11a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 >> 172.21.15.44:0/1954798383 conn(0x7f343c0f76b0 msgr2=0x7f343c0f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 shutdown_connections 2024-04-16T11:55:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:48.668+0000 7f3440c00700 1 -- 172.21.15.44:0/1954798383 wait complete. 2024-04-16T11:55:48.667 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-16T11:55:48.975 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:48 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1954798383' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:55:49.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:48 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1954798383' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T11:55:49.484 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-16T11:55:49.495 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T11:55:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 -- 172.21.15.44:0/3985508510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fdbe0 msgr2=0x7face8106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 --2- 172.21.15.44:0/3985508510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fdbe0 0x7face8106220 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7facd40048d0 tx=0x7facd4009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 -- 172.21.15.44:0/3985508510 shutdown_connections 2024-04-16T11:55:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 --2- 172.21.15.44:0/3985508510 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fdbe0 0x7face8106220 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 --2- 172.21.15.44:0/3985508510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7face80fd240 0x7face80fd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 -- 172.21.15.44:0/3985508510 >> 172.21.15.44:0/3985508510 conn(0x7face80f6cf0 msgr2=0x7face80f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:49.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.725+0000 7faceca00640 1 -- 172.21.15.44:0/3985508510 shutdown_connections 2024-04-16T11:55:49.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.726+0000 7faceca00640 1 -- 172.21.15.44:0/3985508510 wait complete. 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.726+0000 7faceca00640 1 Processor -- start 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.726+0000 7faceca00640 1 -- start start 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7faceca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 0x7face8114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7faceca00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7face80fdbe0 0x7face8115420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7faceca00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face8115a20 con 0x7face80fd240 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7faceca00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face810f070 con 0x7face80fdbe0 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7face5e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 0x7face8114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7face5e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 0x7face8114ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58094/0 (socket says 172.21.15.44:58094) 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7face5e00640 1 -- 172.21.15.44:0/1642297202 learned_addr learned my addr 172.21.15.44:0/1642297202 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:49.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.727+0000 7face5400640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7face80fdbe0 0x7face8115420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7face5e00640 1 -- 172.21.15.44:0/1642297202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7face80fdbe0 msgr2=0x7face8115420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7face5e00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7face80fdbe0 0x7face8115420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7face5e00640 1 -- 172.21.15.44:0/1642297202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4005040 con 0x7face80fd240 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7face5e00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 0x7face8114ee0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7facd800cae0 tx=0x7facd8007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd800aa80 con 0x7face80fd240 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facd800abe0 con 0x7face80fd240 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face810f350 con 0x7face80fd240 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd80116b0 con 0x7face80fd240 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.728+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face810f840 con 0x7face80fd240 2024-04-16T11:55:49.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.730+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face8066070 con 0x7face80fd240 2024-04-16T11:55:49.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.731+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7facd800ad50 con 0x7face80fd240 2024-04-16T11:55:49.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.732+0000 7facde000640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faccc06eb70 0x7faccc071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:49.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.732+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7facd808f400 con 0x7face80fd240 2024-04-16T11:55:49.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.732+0000 7face5400640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faccc06eb70 0x7faccc071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:49.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.733+0000 7face5400640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faccc06eb70 0x7faccc071020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7facd40048a0 tx=0x7facd4009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:49.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.737+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7facd8057eb0 con 0x7face80fd240 2024-04-16T11:55:49.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.944+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7face8064380 con 0x7face80fd240 2024-04-16T11:55:49.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.947+0000 7facde000640 1 -- 172.21.15.44:0/1642297202 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+17517 (secure 0 0 0) 0x7facd805b570 con 0x7face80fd240 2024-04-16T11:55:49.946 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:49.946 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:51:19.474271+0000","last_up_change":"2024-04-16T11:49:35.376417+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T11:39:26.412859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T11:39:27.033592+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":55,"up_thru":85,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6803","nonce":123479871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6805","nonce":123479871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6809","nonce":123479871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6807","nonce":123479871}]},"public_addr":"172.21.15.44:6803/123479871","cluster_addr":"172.21.15.44:6805/123479871","heartbeat_back_addr":"172.21.15.44:6809/123479871","heartbeat_front_addr":"172.21.15.44:6807/123479871","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":57,"up_from":61,"up_thru":85,"down_at":58,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6811","nonce":544437595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6813","nonce":544437595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6817","nonce":544437595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6815","nonce":544437595}]},"public_addr":"172.21.15.44:6811/544437595","cluster_addr":"172.21.15.44:6813/544437595","heartbeat_back_addr":"172.21.15.44:6817/544437595","heartbeat_front_addr":"172.21.15.44:6815/544437595","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":62,"up_from":69,"up_thru":85,"down_at":63,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6819","nonce":4137440489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6821","nonce":4137440489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6825","nonce":4137440489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6823","nonce":4137440489}]},"public_addr":"172.21.15.44:6819/4137440489","cluster_addr":"172.21.15.44:6821/4137440489","heartbeat_back_addr":"172.21.15.44:6825/4137440489","heartbeat_front_addr":"172.21.15.44:6823/4137440489","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":71,"up_from":75,"up_thru":85,"down_at":72,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6801","nonce":2814984900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6803","nonce":2814984900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6807","nonce":2814984900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6805","nonce":2814984900}]},"public_addr":"172.21.15.45:6801/2814984900","cluster_addr":"172.21.15.45:6803/2814984900","heartbeat_back_addr":"172.21.15.45:6807/2814984900","heartbeat_front_addr":"172.21.15.45:6805/2814984900","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":76,"up_from":80,"up_thru":85,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6809","nonce":3529709154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6811","nonce":3529709154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6815","nonce":3529709154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6813","nonce":3529709154}]},"public_addr":"172.21.15.45:6809/3529709154","cluster_addr":"172.21.15.45:6811/3529709154","heartbeat_back_addr":"172.21.15.45:6815/3529709154","heartbeat_front_addr":"172.21.15.45:6813/3529709154","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":81,"up_from":85,"up_thru":85,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6817","nonce":579419871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6819","nonce":579419871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6823","nonce":579419871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6821","nonce":579419871}]},"public_addr":"172.21.15.45:6817/579419871","cluster_addr":"172.21.15.45:6819/579419871","heartbeat_back_addr":"172.21.15.45:6823/579419871","heartbeat_front_addr":"172.21.15.45:6821/579419871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":57},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":62},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":71},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":76},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":81}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":5}]},{"pgid":"2.1","mappings":[{"from":0,"to":5}]},{"pgid":"2.8","mappings":[{"from":3,"to":2},{"from":0,"to":1}]},{"pgid":"2.12","mappings":[{"from":3,"to":2}]},{"pgid":"2.1a","mappings":[{"from":3,"to":2}]},{"pgid":"3.1","mappings":[{"from":0,"to":5}]},{"pgid":"3.d","mappings":[{"from":3,"to":0}]},{"pgid":"3.10","mappings":[{"from":1,"to":2}]},{"pgid":"3.11","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":3,"to":0}]},{"pgid":"3.1c","mappings":[{"from":4,"to":0},{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6827/1837389700":"2024-04-17T11:51:19.474191+0000","172.21.15.45:6826/1837389700":"2024-04-17T11:51:19.474191+0000","172.21.15.45:6825/4128280511":"2024-04-17T11:50:53.137847+0000","172.21.15.45:6824/4128280511":"2024-04-17T11:50:53.137847+0000","172.21.15.44:6829/248184983":"2024-04-17T11:50:21.139145+0000","172.21.15.44:6826/2456403196":"2024-04-17T11:49:47.142427+0000","172.21.15.44:6828/248184983":"2024-04-17T11:50:21.139145+0000","172.21.15.44:0/300728059":"2024-04-17T11:44:40.116622+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6800/2086828317":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3182081166":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/1982927697":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/3788212463":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/3480063073":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3840344332":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/1118320507":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/1480410906":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/1916760620":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6801/2086828317":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.45:6828/1075678648":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/4136480716":"2024-04-17T11:44:40.116622+0000","172.21.15.45:6829/1075678648":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/4260319814":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/2618228669":"2024-04-17T11:43:34.195423+0000","172.21.15.44:6800/3774957652":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/3774957652":"2024-04-17T11:44:40.116622+0000","172.21.15.44:6827/2456403196":"2024-04-17T11:49:47.142427+0000","172.21.15.44:0/567559671":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3726998681":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/1779875453":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/914506740":"2024-04-17T11:44:40.116622+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:55:49.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faccc06eb70 msgr2=0x7faccc071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:49.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faccc06eb70 0x7faccc071020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7facd40048a0 tx=0x7facd4009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 msgr2=0x7face8114ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 0x7face8114ee0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7facd800cae0 tx=0x7facd8007e50 comp rx=0 tx=0).stop 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 shutdown_connections 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faccc06eb70 0x7faccc071020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7face80fdbe0 0x7face8115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 --2- 172.21.15.44:0/1642297202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7face80fd240 0x7face8114ee0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.950+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 >> 172.21.15.44:0/1642297202 conn(0x7face80f6cf0 msgr2=0x7face80fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.951+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 shutdown_connections 2024-04-16T11:55:49.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:49.951+0000 7faceca00640 1 -- 172.21.15.44:0/1642297202 wait complete. 2024-04-16T11:55:49.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T11:36:12.538436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T11:39:26.412859+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T11:39:27.033592+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T11:55:49.967 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T11:55:50.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:49 smithi045 ceph-mon[81423]: pgmap v380: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.187+0000 7f53abe00640 1 -- 172.21.15.44:0/627866566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104250 msgr2=0x7f53ac104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.187+0000 7f53abe00640 1 --2- 172.21.15.44:0/627866566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104250 0x7f53ac104620 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f539c0048d0 tx=0x7f539c009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.188+0000 7f53abe00640 1 -- 172.21.15.44:0/627866566 shutdown_connections 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.188+0000 7f53abe00640 1 --2- 172.21.15.44:0/627866566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac104bf0 0x7f53ac0fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.188+0000 7f53abe00640 1 --2- 172.21.15.44:0/627866566 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104250 0x7f53ac104620 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.188+0000 7f53abe00640 1 -- 172.21.15.44:0/627866566 >> 172.21.15.44:0/627866566 conn(0x7f53ac0f9fc0 msgr2=0x7f53ac0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.188+0000 7f53abe00640 1 -- 172.21.15.44:0/627866566 shutdown_connections 2024-04-16T11:55:50.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.188+0000 7f53abe00640 1 -- 172.21.15.44:0/627866566 wait complete. 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.189+0000 7f53abe00640 1 Processor -- start 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.189+0000 7f53abe00640 1 -- start start 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.189+0000 7f53abe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac104250 0x7f53ac114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.189+0000 7f53abe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 0x7f53ac115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.189+0000 7f53abe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac115880 con 0x7f53ac104bf0 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.189+0000 7f53abe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac10eed0 con 0x7f53ac104250 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53a3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 0x7f53ac115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53a3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 0x7f53ac115280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58126/0 (socket says 172.21.15.44:58126) 2024-04-16T11:55:50.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53a3e00640 1 -- 172.21.15.44:0/2977933727 learned_addr learned my addr 172.21.15.44:0/2977933727 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:50.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53aaa00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac104250 0x7f53ac114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53a3e00640 1 -- 172.21.15.44:0/2977933727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac104250 msgr2=0x7f53ac114d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53a3e00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac104250 0x7f53ac114d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.190+0000 7f53a3e00640 1 -- 172.21.15.44:0/2977933727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c005040 con 0x7f53ac104bf0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.191+0000 7f53a3e00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 0x7f53ac115280 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f539400c810 tx=0x7f53940043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.191+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f539400a820 con 0x7f53ac104bf0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.191+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac10f1b0 con 0x7f53ac104bf0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.191+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f539400ae50 con 0x7f53ac104bf0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.191+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53940115c0 con 0x7f53ac104bf0 2024-04-16T11:55:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.191+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac10f6d0 con 0x7f53ac104bf0 2024-04-16T11:55:50.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.192+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac0ff9c0 con 0x7f53ac104bf0 2024-04-16T11:55:50.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.193+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f539400a980 con 0x7f53ac104bf0 2024-04-16T11:55:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.194+0000 7f53a2a00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f539006eb80 0x7f5390071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.194+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f539408f110 con 0x7f53ac104bf0 2024-04-16T11:55:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.195+0000 7f53aaa00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f539006eb80 0x7f5390071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:50.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.196+0000 7f53aaa00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f539006eb80 0x7f5390071030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f539c031870 tx=0x7f539c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:50.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.199+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5394057c00 con 0x7f53ac104bf0 2024-04-16T11:55:50.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:49 smithi044 ceph-mon[93020]: pgmap v380: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:50.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.405+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53ac1103b0 con 0x7f53ac104bf0 2024-04-16T11:55:50.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.406+0000 7f53a2a00640 1 -- 172.21.15.44:0/2977933727 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f539405b2c0 con 0x7f53ac104bf0 2024-04-16T11:55:50.404 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-16T11:55:50.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f539006eb80 msgr2=0x7f5390071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f539006eb80 0x7f5390071030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f539c031870 tx=0x7f539c009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 msgr2=0x7f53ac115280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 0x7f53ac115280 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f539400c810 tx=0x7f53940043c0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 shutdown_connections 2024-04-16T11:55:50.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f539006eb80 0x7f5390071030 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f53ac104bf0 0x7f53ac115280 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 --2- 172.21.15.44:0/2977933727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac104250 0x7f53ac114d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 >> 172.21.15.44:0/2977933727 conn(0x7f53ac0f9fc0 msgr2=0x7f53ac0fbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:50.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.409+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 shutdown_connections 2024-04-16T11:55:50.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.410+0000 7f53abe00640 1 -- 172.21.15.44:0/2977933727 wait complete. 2024-04-16T11:55:50.427 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 -- 172.21.15.44:0/156872922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104bf0 msgr2=0x7ff2500fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 --2- 172.21.15.44:0/156872922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104bf0 0x7ff2500fe280 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff2380048d0 tx=0x7ff238009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 -- 172.21.15.44:0/156872922 shutdown_connections 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 --2- 172.21.15.44:0/156872922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104bf0 0x7ff2500fe280 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 --2- 172.21.15.44:0/156872922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff250104250 0x7ff250104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 -- 172.21.15.44:0/156872922 >> 172.21.15.44:0/156872922 conn(0x7ff2500f9fc0 msgr2=0x7ff2500fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 -- 172.21.15.44:0/156872922 shutdown_connections 2024-04-16T11:55:50.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.651+0000 7ff254c00640 1 -- 172.21.15.44:0/156872922 wait complete. 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.652+0000 7ff254c00640 1 Processor -- start 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.652+0000 7ff254c00640 1 -- start start 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.652+0000 7ff254c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 0x7ff250114d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.652+0000 7ff254c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff250104bf0 0x7ff2501152b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.652+0000 7ff254c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2501158b0 con 0x7ff250104250 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.652+0000 7ff254c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25010ef00 con 0x7ff250104bf0 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 0x7ff250114d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 0x7ff250114d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58156/0 (socket says 172.21.15.44:58156) 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 -- 172.21.15.44:0/457801069 learned_addr learned my addr 172.21.15.44:0/457801069 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 -- 172.21.15.44:0/457801069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff250104bf0 msgr2=0x7ff2501152b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff250104bf0 0x7ff2501152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 -- 172.21.15.44:0/457801069 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238005040 con 0x7ff250104250 2024-04-16T11:55:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.653+0000 7ff24f400640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 0x7ff250114d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff24000db50 tx=0x7ff240004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.654+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff24000b9d0 con 0x7ff250104250 2024-04-16T11:55:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.654+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25010f1e0 con 0x7ff250104250 2024-04-16T11:55:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.654+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff24000f040 con 0x7ff250104250 2024-04-16T11:55:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.654+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25010f700 con 0x7ff250104250 2024-04-16T11:55:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.654+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff24001d640 con 0x7ff250104250 2024-04-16T11:55:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.655+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2500ff9c0 con 0x7ff250104250 2024-04-16T11:55:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.656+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff240013430 con 0x7ff250104250 2024-04-16T11:55:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.657+0000 7ff247e00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff23406eb80 0x7ff234071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.657+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7ff24008f0d0 con 0x7ff250104250 2024-04-16T11:55:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.657+0000 7ff246a00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff23406eb80 0x7ff234071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.658+0000 7ff246a00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff23406eb80 0x7ff234071030 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff23802bda0 tx=0x7ff238009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:50.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.662+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff240057b80 con 0x7ff250104250 2024-04-16T11:55:50.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.870+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ff25004bcc0 con 0x7ff250104250 2024-04-16T11:55:50.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.871+0000 7ff247e00640 1 -- 172.21.15.44:0/457801069 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v91) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff25004bcc0 con 0x7ff250104250 2024-04-16T11:55:50.870 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:55:50.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff23406eb80 msgr2=0x7ff234071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff23406eb80 0x7ff234071030 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff23802bda0 tx=0x7ff238009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 msgr2=0x7ff250114d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:50.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 0x7ff250114d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff24000db50 tx=0x7ff240004ee0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 shutdown_connections 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7ff23406eb80 0x7ff234071030 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff250104bf0 0x7ff2501152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 --2- 172.21.15.44:0/457801069 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff250104250 0x7ff250114d70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 >> 172.21.15.44:0/457801069 conn(0x7ff2500f9fc0 msgr2=0x7ff2500fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 shutdown_connections 2024-04-16T11:55:50.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:50.874+0000 7ff254c00640 1 -- 172.21.15.44:0/457801069 wait complete. 2024-04-16T11:55:50.891 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T11:55:51.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:50 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1642297202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:55:51.079 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:50 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2977933727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:55:51.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.115+0000 7f942aa00640 1 -- 172.21.15.44:0/588628609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104bf0 msgr2=0x7f94240fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.115+0000 7f942aa00640 1 --2- 172.21.15.44:0/588628609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104bf0 0x7f94240fe280 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f94100048d0 tx=0x7f9410009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.115+0000 7f942aa00640 1 -- 172.21.15.44:0/588628609 shutdown_connections 2024-04-16T11:55:51.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.115+0000 7f942aa00640 1 --2- 172.21.15.44:0/588628609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104bf0 0x7f94240fe280 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.115+0000 7f942aa00640 1 --2- 172.21.15.44:0/588628609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9424104250 0x7f9424104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.115+0000 7f942aa00640 1 -- 172.21.15.44:0/588628609 >> 172.21.15.44:0/588628609 conn(0x7f94240f9fc0 msgr2=0x7f94240fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:51.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.116+0000 7f942aa00640 1 -- 172.21.15.44:0/588628609 shutdown_connections 2024-04-16T11:55:51.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.116+0000 7f942aa00640 1 -- 172.21.15.44:0/588628609 wait complete. 2024-04-16T11:55:51.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.116+0000 7f942aa00640 1 Processor -- start 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f942aa00640 1 -- start start 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f942aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 0x7f9424114d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f942aa00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9424104bf0 0x7f94241152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f942aa00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94241158a0 con 0x7f9424104250 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f942aa00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942410eef0 con 0x7f9424104bf0 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f9429600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 0x7f9424114d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f9429600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 0x7f9424114d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58164/0 (socket says 172.21.15.44:58164) 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.117+0000 7f9429600640 1 -- 172.21.15.44:0/2005298865 learned_addr learned my addr 172.21.15.44:0/2005298865 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9428c00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9424104bf0 0x7f94241152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:51.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9429600640 1 -- 172.21.15.44:0/2005298865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9424104bf0 msgr2=0x7f94241152a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9429600640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9424104bf0 0x7f94241152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9429600640 1 -- 172.21.15.44:0/2005298865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410005040 con 0x7f9424104250 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9429600640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 0x7f9424114d60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9418002a50 tx=0x7f9418004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9418007c30 con 0x7f9424104250 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.118+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f941800f040 con 0x7f9424104250 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.119+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941800db50 con 0x7f9424104250 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.119+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942410f1d0 con 0x7f9424104250 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.119+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942410f6f0 con 0x7f9424104250 2024-04-16T11:55:51.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.120+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94240ff9c0 con 0x7f9424104250 2024-04-16T11:55:51.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.121+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f941800dcf0 con 0x7f9424104250 2024-04-16T11:55:51.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.122+0000 7f9421600640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f940c06eb70 0x7f940c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:51.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.122+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f941808e3e0 con 0x7f9424104250 2024-04-16T11:55:51.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.123+0000 7f9428c00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f940c06eb70 0x7f940c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:51.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.123+0000 7f9428c00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f940c06eb70 0x7f940c071020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9410000c00 tx=0x7f9410009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.127+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9418056f10 con 0x7f9424104250 2024-04-16T11:55:51.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:50 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1642297202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:55:51.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:50 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2977933727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:55:51.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.336+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f942410b3b0 con 0x7f9424104250 2024-04-16T11:55:51.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.337+0000 7f9421600640 1 -- 172.21.15.44:0/2005298865 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v91) v1 ==== 107+0+11 (secure 0 0 0) 0x7f941805a5d0 con 0x7f9424104250 2024-04-16T11:55:51.335 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:55:51.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f940c06eb70 msgr2=0x7f940c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f940c06eb70 0x7f940c071020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9410000c00 tx=0x7f9410009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 msgr2=0x7f9424114d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 0x7f9424114d60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9418002a50 tx=0x7f9418004600 comp rx=0 tx=0).stop 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 shutdown_connections 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f940c06eb70 0x7f940c071020 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9424104bf0 0x7f94241152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 --2- 172.21.15.44:0/2005298865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9424104250 0x7f9424114d60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 >> 172.21.15.44:0/2005298865 conn(0x7f94240f9fc0 msgr2=0x7f94240fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 shutdown_connections 2024-04-16T11:55:51.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.340+0000 7f942aa00640 1 -- 172.21.15.44:0/2005298865 wait complete. 2024-04-16T11:55:51.357 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T11:55:51.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 -- 172.21.15.44:0/1374671954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd81041f0 msgr2=0x7f8bd81045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/1374671954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd81041f0 0x7f8bd81045c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009c50 tx=0x7f8bc8004d10 comp rx=0 tx=0).stop 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 -- 172.21.15.44:0/1374671954 shutdown_connections 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/1374671954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8104b90 0x7f8bd80fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/1374671954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd81041f0 0x7f8bd81045c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 -- 172.21.15.44:0/1374671954 >> 172.21.15.44:0/1374671954 conn(0x7f8bd80f9fc0 msgr2=0x7f8bd80fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 -- 172.21.15.44:0/1374671954 shutdown_connections 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.571+0000 7f8bd7e00640 1 -- 172.21.15.44:0/1374671954 wait complete. 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd7e00640 1 Processor -- start 2024-04-16T11:55:51.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd7e00640 1 -- start start 2024-04-16T11:55:51.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd81041f0 0x7f8bd8114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:51.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 0x7f8bd81151d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:51.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd7e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd81157d0 con 0x7f8bd8104b90 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd810ee20 con 0x7f8bd81041f0 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd6000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 0x7f8bd81151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd6000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 0x7f8bd81151d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53940/0 (socket says 172.21.15.44:53940) 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.572+0000 7f8bd6000640 1 -- 172.21.15.44:0/292425824 learned_addr learned my addr 172.21.15.44:0/292425824 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.573+0000 7f8bd6a00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd81041f0 0x7f8bd8114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.573+0000 7f8bd6000640 1 -- 172.21.15.44:0/292425824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd81041f0 msgr2=0x7f8bd8114c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.573+0000 7f8bd6000640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd81041f0 0x7f8bd8114c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.573+0000 7f8bd6000640 1 -- 172.21.15.44:0/292425824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80098b0 con 0x7f8bd8104b90 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.573+0000 7f8bd6000640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 0x7f8bd81151d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8bc000c830 tx=0x7f8bc0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.574+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bc000a850 con 0x7f8bd8104b90 2024-04-16T11:55:51.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.574+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bc000ae80 con 0x7f8bd8104b90 2024-04-16T11:55:51.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.574+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bc0012ea0 con 0x7f8bd8104b90 2024-04-16T11:55:51.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.574+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd810f100 con 0x7f8bd8104b90 2024-04-16T11:55:51.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.574+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd810f6a0 con 0x7f8bd8104b90 2024-04-16T11:55:51.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.575+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd80ff960 con 0x7f8bd8104b90 2024-04-16T11:55:51.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.576+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8bc0012400 con 0x7f8bd8104b90 2024-04-16T11:55:51.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.577+0000 7f8bcea00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8bbc06eb80 0x7f8bbc071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:51.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.577+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f8bc008ec60 con 0x7f8bd8104b90 2024-04-16T11:55:51.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.577+0000 7f8bd6a00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8bbc06eb80 0x7f8bbc071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:51.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.578+0000 7f8bd6a00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8bbc06eb80 0x7f8bbc071030 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8004a30 tx=0x7f8bc802e040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:51.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.582+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8bc00578d0 con 0x7f8bd8104b90 2024-04-16T11:55:51.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.830+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bd804bcc0 con 0x7f8bd8104b90 2024-04-16T11:55:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.832+0000 7f8bcea00640 1 -- 172.21.15.44:0/292425824 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5201 (secure 0 0 0) 0x7f8bc00126b0 con 0x7f8bd8104b90 2024-04-16T11:55:51.830 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:51.830 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44319,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6825/33255321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":33255321},{"type":"v1","addr":"172.21.15.45:6825","nonce":33255321}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44341,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6827/3167580677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3167580677},{"type":"v1","addr":"172.21.15.45:6827","nonce":3167580677}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:51:22.505053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34432":{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":0,"incarnation":33,"state":"up:active","state_seq":24,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34432,"qdb_cluster":[34432]},"id":1}]} 2024-04-16T11:55:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 37 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8bbc06eb80 msgr2=0x7f8bbc071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8bbc06eb80 0x7f8bbc071030 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8004a30 tx=0x7f8bc802e040 comp rx=0 tx=0).stop 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 msgr2=0x7f8bd81151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 0x7f8bd81151d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8bc000c830 tx=0x7f8bc0004430 comp rx=0 tx=0).stop 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 shutdown_connections 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f8bbc06eb80 0x7f8bbc071030 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bd8104b90 0x7f8bd81151d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 --2- 172.21.15.44:0/292425824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd81041f0 0x7f8bd8114c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 >> 172.21.15.44:0/292425824 conn(0x7f8bd80f9fc0 msgr2=0x7f8bd80fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:51.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 shutdown_connections 2024-04-16T11:55:51.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:51.835+0000 7f8bd7e00640 1 -- 172.21.15.44:0/292425824 wait complete. 2024-04-16T11:55:51.852 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 12, 'max_mds': 1, 'flags': 18} 2024-04-16T11:55:51.853 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 13 2024-04-16T11:55:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 -- 172.21.15.44:0/2505836807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 msgr2=0x7fa54c0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 --2- 172.21.15.44:0/2505836807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c0fc250 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa5380048d0 tx=0x7fa538009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 -- 172.21.15.44:0/2505836807 shutdown_connections 2024-04-16T11:55:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 --2- 172.21.15.44:0/2505836807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c0fc250 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 --2- 172.21.15.44:0/2505836807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa54c102220 0x7fa54c1025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 -- 172.21.15.44:0/2505836807 >> 172.21.15.44:0/2505836807 conn(0x7fa54c0f7e10 msgr2=0x7fa54c0fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 -- 172.21.15.44:0/2505836807 shutdown_connections 2024-04-16T11:55:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.073+0000 7fa551000640 1 -- 172.21.15.44:0/2505836807 wait complete. 2024-04-16T11:55:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.074+0000 7fa551000640 1 Processor -- start 2024-04-16T11:55:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.074+0000 7fa551000640 1 -- start start 2024-04-16T11:55:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa551000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa54c102220 0x7fa54c114ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa551000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c115400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa551000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c115a00 con 0x7fa54c102bc0 2024-04-16T11:55:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa551000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c10f0a0 con 0x7fa54c102220 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa549400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c115400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa549400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c115400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53966/0 (socket says 172.21.15.44:53966) 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa549400640 1 -- 172.21.15.44:0/4025495990 learned_addr learned my addr 172.21.15.44:0/4025495990 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.075+0000 7fa549e00640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa54c102220 0x7fa54c114ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa549400640 1 -- 172.21.15.44:0/4025495990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa54c102220 msgr2=0x7fa54c114ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa549400640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa54c102220 0x7fa54c114ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa549400640 1 -- 172.21.15.44:0/4025495990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538005040 con 0x7fa54c102bc0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa549400640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c115400 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5380047f0 tx=0x7fa538009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5380094c0 con 0x7fa54c102bc0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa538032e50 con 0x7fa54c102bc0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c10f320 con 0x7fa54c102bc0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5380433b0 con 0x7fa54c102bc0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.076+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c10f7e0 con 0x7fa54c102bc0 2024-04-16T11:55:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.078+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c0fd990 con 0x7fa54c102bc0 2024-04-16T11:55:52.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.079+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa53802f070 con 0x7fa54c102bc0 2024-04-16T11:55:52.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.080+0000 7fa542000640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa53006eb70 0x7fa530071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:52.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.080+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fa5380b0af0 con 0x7fa54c102bc0 2024-04-16T11:55:52.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.080+0000 7fa549e00640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa53006eb70 0x7fa530071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:52.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.081+0000 7fa549e00640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa53006eb70 0x7fa530071020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa53c007fd0 tx=0x7fa53c009e90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:52.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.084+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa5380795a0 con 0x7fa54c102bc0 2024-04-16T11:55:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:51 smithi044 ceph-mon[93020]: pgmap v381: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:51 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/457801069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:55:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:51 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2005298865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:55:52.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:51 smithi045 ceph-mon[81423]: pgmap v381: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:51 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/457801069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:55:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:51 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2005298865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:55:52.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.333+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 13, "format": "json"} v 0) v1 -- 0x7fa54c1093b0 con 0x7fa54c102bc0 2024-04-16T11:55:52.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.335+0000 7fa542000640 1 -- 172.21.15.44:0/4025495990 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 13, "format": "json"}]=0 dumped fsmap epoch 13 v37) v1 ==== 107+0+4852 (secure 0 0 0) 0x7fa53807cc60 con 0x7fa54c102bc0 2024-04-16T11:55:52.333 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:52.333 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:39:47.009160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14456},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14456":{"gid":14456,"name":"cephfs.smithi044.ufhhlc","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.44:6827/2456403196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2456403196},{"type":"v1","addr":"172.21.15.44:6827","nonce":2456403196}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:52.334 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-16T11:55:52.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa53006eb70 msgr2=0x7fa530071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa53006eb70 0x7fa530071020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa53c007fd0 tx=0x7fa53c009e90 comp rx=0 tx=0).stop 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 msgr2=0x7fa54c115400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c115400 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5380047f0 tx=0x7fa538009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 shutdown_connections 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa53006eb70 0x7fa530071020 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa54c102bc0 0x7fa54c115400 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 --2- 172.21.15.44:0/4025495990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa54c102220 0x7fa54c114ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 >> 172.21.15.44:0/4025495990 conn(0x7fa54c0f7e10 msgr2=0x7fa54c101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:52.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.338+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 shutdown_connections 2024-04-16T11:55:52.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.339+0000 7fa551000640 1 -- 172.21.15.44:0/4025495990 wait complete. 2024-04-16T11:55:52.355 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 14 2024-04-16T11:55:52.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 -- 172.21.15.44:0/4183613911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104bd0 msgr2=0x7fec280fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 --2- 172.21.15.44:0/4183613911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104bd0 0x7fec280fe260 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fec140048d0 tx=0x7fec14009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 -- 172.21.15.44:0/4183613911 shutdown_connections 2024-04-16T11:55:52.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 --2- 172.21.15.44:0/4183613911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104bd0 0x7fec280fe260 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 --2- 172.21.15.44:0/4183613911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104230 0x7fec28104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 -- 172.21.15.44:0/4183613911 >> 172.21.15.44:0/4183613911 conn(0x7fec280f9fc0 msgr2=0x7fec280fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:52.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 -- 172.21.15.44:0/4183613911 shutdown_connections 2024-04-16T11:55:52.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.574+0000 7fec2e200640 1 -- 172.21.15.44:0/4183613911 wait complete. 2024-04-16T11:55:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.575+0000 7fec2e200640 1 Processor -- start 2024-04-16T11:55:52.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.575+0000 7fec2e200640 1 -- start start 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2e200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 0x7fec28114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2e200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104bd0 0x7fec28115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2e200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec28115860 con 0x7fec28104230 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2e200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2810eeb0 con 0x7fec28104bd0 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2ce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 0x7fec28114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2ce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 0x7fec28114d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53986/0 (socket says 172.21.15.44:53986) 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2ce00640 1 -- 172.21.15.44:0/2444602567 learned_addr learned my addr 172.21.15.44:0/2444602567 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2ce00640 1 -- 172.21.15.44:0/2444602567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104bd0 msgr2=0x7fec28115260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec27e00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104bd0 0x7fec28115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2ce00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104bd0 0x7fec28115260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec2ce00640 1 -- 172.21.15.44:0/2444602567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec14005040 con 0x7fec28104230 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.576+0000 7fec27e00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104bd0 0x7fec28115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.577+0000 7fec2ce00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 0x7fec28114d20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec1c002a50 tx=0x7fec1c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.577+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1c007c30 con 0x7fec28104230 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.577+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec1c00f040 con 0x7fec28104230 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.577+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2810f190 con 0x7fec28104230 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.577+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1c00db50 con 0x7fec28104230 2024-04-16T11:55:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.577+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2810f6b0 con 0x7fec28104230 2024-04-16T11:55:52.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.578+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec280ff9a0 con 0x7fec28104230 2024-04-16T11:55:52.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.579+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec1c00dcf0 con 0x7fec28104230 2024-04-16T11:55:52.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.580+0000 7fec24c00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fec1006eb70 0x7fec10071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:52.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.580+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fec1c08e3e0 con 0x7fec28104230 2024-04-16T11:55:52.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.580+0000 7fec27e00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fec1006eb70 0x7fec10071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:52.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.581+0000 7fec27e00640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fec1006eb70 0x7fec10071020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fec14000c00 tx=0x7fec14009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:52.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.585+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fec1c056f10 con 0x7fec28104230 2024-04-16T11:55:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.836+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 14, "format": "json"} v 0) v1 -- 0x7fec281103f0 con 0x7fec28104230 2024-04-16T11:55:52.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.841+0000 7fec24c00640 1 -- 172.21.15.44:0/2444602567 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 14, "format": "json"}]=0 dumped fsmap epoch 14 v37) v1 ==== 107+0+4064 (secure 0 0 0) 0x7fec1c05a5d0 con 0x7fec28104230 2024-04-16T11:55:52.840 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:52.840 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:49:47.142449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 14 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.844+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fec1006eb70 msgr2=0x7fec10071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.844+0000 7fec2e200640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fec1006eb70 0x7fec10071020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fec14000c00 tx=0x7fec14009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.844+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 msgr2=0x7fec28114d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.844+0000 7fec2e200640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 0x7fec28114d20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec1c002a50 tx=0x7fec1c004600 comp rx=0 tx=0).stop 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 shutdown_connections 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fec1006eb70 0x7fec10071020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec28104bd0 0x7fec28115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 --2- 172.21.15.44:0/2444602567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fec28104230 0x7fec28114d20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 >> 172.21.15.44:0/2444602567 conn(0x7fec280f9fc0 msgr2=0x7fec280fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:52.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 shutdown_connections 2024-04-16T11:55:52.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:52.845+0000 7fec2e200640 1 -- 172.21.15.44:0/2444602567 wait complete. 2024-04-16T11:55:52.860 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 15 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 -- 172.21.15.44:0/1312907783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074071a00 msgr2=0x7fe074071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 --2- 172.21.15.44:0/1312907783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074071a00 0x7fe074071dd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe0680048d0 tx=0x7fe068009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 -- 172.21.15.44:0/1312907783 shutdown_connections 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 --2- 172.21.15.44:0/1312907783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe074072310 0x7fe07410cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 --2- 172.21.15.44:0/1312907783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074071a00 0x7fe074071dd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 -- 172.21.15.44:0/1312907783 >> 172.21.15.44:0/1312907783 conn(0x7fe0740f9ff0 msgr2=0x7fe0740fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 -- 172.21.15.44:0/1312907783 shutdown_connections 2024-04-16T11:55:53.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.074+0000 7fe07c800640 1 -- 172.21.15.44:0/1312907783 wait complete. 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe07c800640 1 Processor -- start 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe07c800640 1 -- start start 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe07c800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe074071a00 0x7fe0741192e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe07c800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 0x7fe074119820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe07c800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe074119e20 con 0x7fe074072310 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe07c800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe074113470 con 0x7fe074071a00 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.075+0000 7fe079400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 0x7fe074119820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 0x7fe074119820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54000/0 (socket says 172.21.15.44:54000) 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079400640 1 -- 172.21.15.44:0/1428948809 learned_addr learned my addr 172.21.15.44:0/1428948809 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079e00640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe074071a00 0x7fe0741192e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:53.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079400640 1 -- 172.21.15.44:0/1428948809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe074071a00 msgr2=0x7fe0741192e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079400640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe074071a00 0x7fe0741192e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079400640 1 -- 172.21.15.44:0/1428948809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068005040 con 0x7fe074072310 2024-04-16T11:55:53.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe079400640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 0x7fe074119820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c830 tx=0x7fe064004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:53.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.076+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe06400a850 con 0x7fe074072310 2024-04-16T11:55:53.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.077+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe06400ae80 con 0x7fe074072310 2024-04-16T11:55:53.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.077+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe064012ea0 con 0x7fe074072310 2024-04-16T11:55:53.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.077+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe074113750 con 0x7fe074072310 2024-04-16T11:55:53.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.077+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe074113c70 con 0x7fe074072310 2024-04-16T11:55:53.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.078+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe074073140 con 0x7fe074072310 2024-04-16T11:55:53.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.079+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe064012400 con 0x7fe074072310 2024-04-16T11:55:53.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.080+0000 7fe06e000640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe05c06eb70 0x7fe05c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:53.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.080+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fe06408ee20 con 0x7fe074072310 2024-04-16T11:55:53.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.080+0000 7fe079e00640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe05c06eb70 0x7fe05c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:53.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.081+0000 7fe079e00640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe05c06eb70 0x7fe05c071020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe068031870 tx=0x7fe068009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:53.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.085+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe064057950 con 0x7fe074072310 2024-04-16T11:55:53.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:52 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/292425824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T11:55:53.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:52 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/4025495990' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 13, "format": "json"}]: dispatch 2024-04-16T11:55:53.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:52 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/292425824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T11:55:53.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:52 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/4025495990' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 13, "format": "json"}]: dispatch 2024-04-16T11:55:53.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.332+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 15, "format": "json"} v 0) v1 -- 0x7fe07410f950 con 0x7fe074072310 2024-04-16T11:55:53.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.333+0000 7fe06e000640 1 -- 172.21.15.44:0/1428948809 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 15, "format": "json"}]=0 dumped fsmap epoch 15 v37) v1 ==== 107+0+4075 (secure 0 0 0) 0x7fe0640126b0 con 0x7fe074072310 2024-04-16T11:55:53.331 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:53.331 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:49:47.152800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":0,"incarnation":15,"state":"up:replay","state_seq":2,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:53.332 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 15 2024-04-16T11:55:53.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe05c06eb70 msgr2=0x7fe05c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe05c06eb70 0x7fe05c071020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe068031870 tx=0x7fe068009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 msgr2=0x7fe074119820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 0x7fe074119820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c830 tx=0x7fe064004430 comp rx=0 tx=0).stop 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 shutdown_connections 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe05c06eb70 0x7fe05c071020 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe074072310 0x7fe074119820 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 --2- 172.21.15.44:0/1428948809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe074071a00 0x7fe0741192e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 >> 172.21.15.44:0/1428948809 conn(0x7fe0740f9ff0 msgr2=0x7fe074106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.336+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 shutdown_connections 2024-04-16T11:55:53.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.337+0000 7fe07c800640 1 -- 172.21.15.44:0/1428948809 wait complete. 2024-04-16T11:55:53.354 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-04-16T11:55:53.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 -- 172.21.15.44:0/1696057960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 msgr2=0x7f073c102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 --2- 172.21.15.44:0/1696057960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c102ae0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f07300048d0 tx=0x7f0730009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 -- 172.21.15.44:0/1696057960 shutdown_connections 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 --2- 172.21.15.44:0/1696057960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c102ae0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 --2- 172.21.15.44:0/1696057960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c1075a0 0x7f073c107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 -- 172.21.15.44:0/1696057960 >> 172.21.15.44:0/1696057960 conn(0x7f073c071e60 msgr2=0x7f073c074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 -- 172.21.15.44:0/1696057960 shutdown_connections 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.575+0000 7f0742a00640 1 -- 172.21.15.44:0/1696057960 wait complete. 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0742a00640 1 Processor -- start 2024-04-16T11:55:53.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0742a00640 1 -- start start 2024-04-16T11:55:53.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0742a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c119110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:53.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0742a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c1075a0 0x7f073c119650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:53.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0742a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c119c50 con 0x7f073c102670 2024-04-16T11:55:53.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0742a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c1132f0 con 0x7f073c1075a0 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0741600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c119110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0741600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c119110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54020/0 (socket says 172.21.15.44:54020) 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.576+0000 7f0741600640 1 -- 172.21.15.44:0/1663408478 learned_addr learned my addr 172.21.15.44:0/1663408478 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0740c00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c1075a0 0x7f073c119650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0741600640 1 -- 172.21.15.44:0/1663408478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c1075a0 msgr2=0x7f073c119650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0741600640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c1075a0 0x7f073c119650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0741600640 1 -- 172.21.15.44:0/1663408478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0730005040 con 0x7f073c102670 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0741600640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c119110 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f072c002840 tx=0x7f072c004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072c007c30 con 0x7f073c102670 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f072c00f040 con 0x7f073c102670 2024-04-16T11:55:53.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072c009590 con 0x7f073c102670 2024-04-16T11:55:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.577+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c1135d0 con 0x7f073c102670 2024-04-16T11:55:53.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.578+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c113af0 con 0x7f073c102670 2024-04-16T11:55:53.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.579+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c103d60 con 0x7f073c102670 2024-04-16T11:55:53.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.580+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f072c00d6b0 con 0x7f073c102670 2024-04-16T11:55:53.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.581+0000 7f0739600640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f072406eb70 0x7f0724071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:53.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.581+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f072c08ea10 con 0x7f073c102670 2024-04-16T11:55:53.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.581+0000 7f0740c00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f072406eb70 0x7f0724071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:53.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.582+0000 7f0740c00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f072406eb70 0x7f0724071020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0730000c00 tx=0x7f0730009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:53.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.586+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f072c057540 con 0x7f073c102670 2024-04-16T11:55:53.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.834+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7f073c114810 con 0x7f073c102670 2024-04-16T11:55:53.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.836+0000 7f0739600640 1 -- 172.21.15.44:0/1663408478 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v37) v1 ==== 107+0+4080 (secure 0 0 0) 0x7f072c05ac00 con 0x7f073c102670 2024-04-16T11:55:53.834 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:53.834 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:49:47.183847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":153,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:53.834 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 16 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f072406eb70 msgr2=0x7f0724071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f072406eb70 0x7f0724071020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0730000c00 tx=0x7f0730009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 msgr2=0x7f073c119110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c119110 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f072c002840 tx=0x7f072c004570 comp rx=0 tx=0).stop 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 shutdown_connections 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f072406eb70 0x7f0724071020 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f073c1075a0 0x7f073c119650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 --2- 172.21.15.44:0/1663408478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f073c102670 0x7f073c119110 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 >> 172.21.15.44:0/1663408478 conn(0x7f073c071e60 msgr2=0x7f073c104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 shutdown_connections 2024-04-16T11:55:53.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:53.839+0000 7f0742a00640 1 -- 172.21.15.44:0/1663408478 wait complete. 2024-04-16T11:55:53.856 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-04-16T11:55:54.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 -- 172.21.15.44:0/741968410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc072310 msgr2=0x7f30fc10c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 --2- 172.21.15.44:0/741968410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc072310 0x7f30fc10c9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009c50 tx=0x7f30ec004d10 comp rx=0 tx=0).stop 2024-04-16T11:55:54.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 -- 172.21.15.44:0/741968410 shutdown_connections 2024-04-16T11:55:54.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 --2- 172.21.15.44:0/741968410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc072310 0x7f30fc10c9a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 --2- 172.21.15.44:0/741968410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30fc071a00 0x7f30fc071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 -- 172.21.15.44:0/741968410 >> 172.21.15.44:0/741968410 conn(0x7f30fc0f9fc0 msgr2=0x7f30fc0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:54.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 -- 172.21.15.44:0/741968410 shutdown_connections 2024-04-16T11:55:54.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.068+0000 7f3103000640 1 -- 172.21.15.44:0/741968410 wait complete. 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.069+0000 7f3103000640 1 Processor -- start 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.069+0000 7f3103000640 1 -- start start 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.069+0000 7f3103000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 0x7f30fc1190e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.069+0000 7f3103000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30fc072310 0x7f30fc119620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.069+0000 7f3103000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc119c20 con 0x7f30fc071a00 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.069+0000 7f3103000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc113270 con 0x7f30fc072310 2024-04-16T11:55:54.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 0x7f30fc1190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 0x7f30fc1190e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54040/0 (socket says 172.21.15.44:54040) 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101c00640 1 -- 172.21.15.44:0/348980441 learned_addr learned my addr 172.21.15.44:0/348980441 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101200640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30fc072310 0x7f30fc119620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101c00640 1 -- 172.21.15.44:0/348980441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30fc072310 msgr2=0x7f30fc119620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101c00640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30fc072310 0x7f30fc119620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.070+0000 7f3101c00640 1 -- 172.21.15.44:0/348980441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec0098b0 con 0x7f30fc071a00 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.071+0000 7f3101c00640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 0x7f30fc1190e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c810 tx=0x7f30f00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.071+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30f000a820 con 0x7f30fc071a00 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.071+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30f000ae50 con 0x7f30fc071a00 2024-04-16T11:55:54.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.071+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30fc113550 con 0x7f30fc071a00 2024-04-16T11:55:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.071+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30f00115c0 con 0x7f30fc071a00 2024-04-16T11:55:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.071+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc113af0 con 0x7f30fc071a00 2024-04-16T11:55:54.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.072+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc073140 con 0x7f30fc071a00 2024-04-16T11:55:54.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.074+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f30f000a980 con 0x7f30fc071a00 2024-04-16T11:55:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.075+0000 7f30f9600640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f30e406eb70 0x7f30e4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.075+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f30f008f150 con 0x7f30fc071a00 2024-04-16T11:55:54.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.075+0000 7f3101200640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f30e406eb70 0x7f30e4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:54.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.076+0000 7f3101200640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f30e406eb70 0x7f30e4071020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f30fc1149a0 tx=0x7f30ec02e040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:54.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.080+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f30f0057c00 con 0x7f30fc071a00 2024-04-16T11:55:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:53 smithi044 ceph-mon[93020]: pgmap v382: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:53 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2444602567' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 14, "format": "json"}]: dispatch 2024-04-16T11:55:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:53 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1428948809' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-04-16T11:55:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:53 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1663408478' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-16T11:55:54.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.326+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f30fc10f760 con 0x7f30fc071a00 2024-04-16T11:55:54.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.327+0000 7f30f9600640 1 -- 172.21.15.44:0/348980441 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v37) v1 ==== 107+0+4077 (secure 0 0 0) 0x7f30f005b2c0 con 0x7f30fc071a00 2024-04-16T11:55:54.325 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:54.325 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:49:48.166946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":154,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:54.326 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2024-04-16T11:55:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f30e406eb70 msgr2=0x7f30e4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f30e406eb70 0x7f30e4071020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f30fc1149a0 tx=0x7f30ec02e040 comp rx=0 tx=0).stop 2024-04-16T11:55:54.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 msgr2=0x7f30fc1190e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 0x7f30fc1190e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c810 tx=0x7f30f00043c0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 shutdown_connections 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f30e406eb70 0x7f30e4071020 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30fc072310 0x7f30fc119620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 --2- 172.21.15.44:0/348980441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30fc071a00 0x7f30fc1190e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 >> 172.21.15.44:0/348980441 conn(0x7f30fc0f9fc0 msgr2=0x7f30fc0fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 shutdown_connections 2024-04-16T11:55:54.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.330+0000 7f3103000640 1 -- 172.21.15.44:0/348980441 wait complete. 2024-04-16T11:55:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:53 smithi045 ceph-mon[81423]: pgmap v382: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:53 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2444602567' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 14, "format": "json"}]: dispatch 2024-04-16T11:55:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:53 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1428948809' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-04-16T11:55:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:53 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1663408478' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-16T11:55:54.346 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-04-16T11:55:54.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 -- 172.21.15.44:0/925870054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b301075a0 msgr2=0x7f3b30107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 --2- 172.21.15.44:0/925870054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b301075a0 0x7f3b30107970 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3b200048d0 tx=0x7f3b20009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 -- 172.21.15.44:0/925870054 shutdown_connections 2024-04-16T11:55:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 --2- 172.21.15.44:0/925870054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b30102670 0x7f3b30102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 --2- 172.21.15.44:0/925870054 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b301075a0 0x7f3b30107970 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 -- 172.21.15.44:0/925870054 >> 172.21.15.44:0/925870054 conn(0x7f3b30071e60 msgr2=0x7f3b30074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 -- 172.21.15.44:0/925870054 shutdown_connections 2024-04-16T11:55:54.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.551+0000 7f3b34c00640 1 -- 172.21.15.44:0/925870054 wait complete. 2024-04-16T11:55:54.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.552+0000 7f3b34c00640 1 Processor -- start 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.552+0000 7f3b34c00640 1 -- start start 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.552+0000 7f3b34c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 0x7f3b30119170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.552+0000 7f3b34c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b301075a0 0x7f3b301196b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.552+0000 7f3b34c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30119cb0 con 0x7f3b30102670 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.552+0000 7f3b34c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30113300 con 0x7f3b301075a0 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 0x7f3b30119170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 0x7f3b30119170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54054/0 (socket says 172.21.15.44:54054) 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b2f400640 1 -- 172.21.15.44:0/3619317838 learned_addr learned my addr 172.21.15.44:0/3619317838 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b27e00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b301075a0 0x7f3b301196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:54.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b2f400640 1 -- 172.21.15.44:0/3619317838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b301075a0 msgr2=0x7f3b301196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b2f400640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b301075a0 0x7f3b301196b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b2f400640 1 -- 172.21.15.44:0/3619317838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20005040 con 0x7f3b30102670 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.553+0000 7f3b27e00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b301075a0 0x7f3b301196b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.554+0000 7f3b2f400640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 0x7f3b30119170 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3b2002be70 tx=0x7f3b20009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.554+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b200094c0 con 0x7f3b30102670 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.554+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b20032e70 con 0x7f3b30102670 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.554+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b200433b0 con 0x7f3b30102670 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.554+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b30113580 con 0x7f3b30102670 2024-04-16T11:55:54.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.554+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b30113a40 con 0x7f3b30102670 2024-04-16T11:55:54.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.556+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b30103d60 con 0x7f3b30102670 2024-04-16T11:55:54.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.557+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3b2002f070 con 0x7f3b30102670 2024-04-16T11:55:54.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.557+0000 7f3b27400640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3b1406eb70 0x7f3b14071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:54.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.558+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f3b200b0940 con 0x7f3b30102670 2024-04-16T11:55:54.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.558+0000 7f3b27e00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3b1406eb70 0x7f3b14071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:54.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.559+0000 7f3b27e00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3b1406eb70 0x7f3b14071020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b30114a30 tx=0x7f3b1800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:54.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.562+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3b200793f0 con 0x7f3b30102670 2024-04-16T11:55:54.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.806+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7f3b30114810 con 0x7f3b30102670 2024-04-16T11:55:54.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.807+0000 7f3b27400640 1 -- 172.21.15.44:0/3619317838 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v37) v1 ==== 107+0+4086 (secure 0 0 0) 0x7f3b2007cab0 con 0x7f3b30102670 2024-04-16T11:55:54.805 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:54.805 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:49:50.173765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":0,"incarnation":15,"state":"up:active","state_seq":155,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14470,"qdb_cluster":[14470]},"id":1}]} 2024-04-16T11:55:54.805 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 18 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3b1406eb70 msgr2=0x7f3b14071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3b1406eb70 0x7f3b14071020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b30114a30 tx=0x7f3b1800b040 comp rx=0 tx=0).stop 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 msgr2=0x7f3b30119170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 0x7f3b30119170 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3b2002be70 tx=0x7f3b20009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 shutdown_connections 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f3b1406eb70 0x7f3b14071020 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b301075a0 0x7f3b301196b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 --2- 172.21.15.44:0/3619317838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3b30102670 0x7f3b30119170 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 >> 172.21.15.44:0/3619317838 conn(0x7f3b30071e60 msgr2=0x7f3b30104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:54.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 shutdown_connections 2024-04-16T11:55:54.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:54.810+0000 7f3b34c00640 1 -- 172.21.15.44:0/3619317838 wait complete. 2024-04-16T11:55:54.827 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 -- 172.21.15.44:0/158315746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6640104d60 msgr2=0x7f66400fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 --2- 172.21.15.44:0/158315746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6640104d60 0x7f66400fe3f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f66280048d0 tx=0x7f6628009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 -- 172.21.15.44:0/158315746 shutdown_connections 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 --2- 172.21.15.44:0/158315746 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6640104d60 0x7f66400fe3f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 --2- 172.21.15.44:0/158315746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66401043c0 0x7f6640104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 -- 172.21.15.44:0/158315746 >> 172.21.15.44:0/158315746 conn(0x7f66400f9fc0 msgr2=0x7f66400fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:55.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 -- 172.21.15.44:0/158315746 shutdown_connections 2024-04-16T11:55:55.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.039+0000 7f6646e00640 1 -- 172.21.15.44:0/158315746 wait complete. 2024-04-16T11:55:55.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.040+0000 7f6646e00640 1 Processor -- start 2024-04-16T11:55:55.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.040+0000 7f6646e00640 1 -- start start 2024-04-16T11:55:55.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.040+0000 7f6646e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 0x7f6640114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.040+0000 7f6646e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640104d60 0x7f66401153f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.040+0000 7f6646e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401159f0 con 0x7f66401043c0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.040+0000 7f6646e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664010f040 con 0x7f6640104d60 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 0x7f6640114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 0x7f6640114eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54070/0 (socket says 172.21.15.44:54070) 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 -- 172.21.15.44:0/1618486277 learned_addr learned my addr 172.21.15.44:0/1618486277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f6637400640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640104d60 0x7f66401153f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 -- 172.21.15.44:0/1618486277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640104d60 msgr2=0x7f66401153f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640104d60 0x7f66401153f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 -- 172.21.15.44:0/1618486277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6628005040 con 0x7f66401043c0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f6637400640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640104d60 0x7f66401153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.041+0000 7f663fe00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 0x7f6640114eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f663000cae0 tx=0x7f6630007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.042+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663000ab00 con 0x7f66401043c0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.042+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f663000ac60 con 0x7f66401043c0 2024-04-16T11:55:55.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.042+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66300116f0 con 0x7f66401043c0 2024-04-16T11:55:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.042+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664010f320 con 0x7f66401043c0 2024-04-16T11:55:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.042+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664010f840 con 0x7f66401043c0 2024-04-16T11:55:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.043+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66400ffb30 con 0x7f66401043c0 2024-04-16T11:55:55.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.044+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6630012440 con 0x7f66401043c0 2024-04-16T11:55:55.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.045+0000 7f663cc00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f662406eb70 0x7f6624071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:55.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.045+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f663008f3d0 con 0x7f66401043c0 2024-04-16T11:55:55.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.045+0000 7f6637400640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f662406eb70 0x7f6624071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:55.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.046+0000 7f6637400640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f662406eb70 0x7f6624071020 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6628000c00 tx=0x7f6628009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:55.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.050+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6630057f00 con 0x7f66401043c0 2024-04-16T11:55:55.128 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:54 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/348980441' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-16T11:55:55.128 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:54 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3619317838' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-16T11:55:55.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.300+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f6640064380 con 0x7f66401043c0 2024-04-16T11:55:55.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.301+0000 7f663cc00640 1 -- 172.21.15.44:0/1618486277 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v37) v1 ==== 107+0+4933 (secure 0 0 0) 0x7f663005b5c0 con 0x7f66401043c0 2024-04-16T11:55:55.299 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:55.300 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:49:50.173765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14470":{"gid":14470,"name":"cephfs.smithi044.hlafhv","rank":0,"incarnation":15,"state":"up:active","state_seq":155,"addr":"172.21.15.44:6829/248184983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":248184983},{"type":"v1","addr":"172.21.15.44:6829","nonce":248184983}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14470,"qdb_cluster":[14470]},"id":1}]} 2024-04-16T11:55:55.300 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 19 2024-04-16T11:55:55.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f662406eb70 msgr2=0x7f6624071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f662406eb70 0x7f6624071020 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6628000c00 tx=0x7f6628009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 msgr2=0x7f6640114eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 0x7f6640114eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f663000cae0 tx=0x7f6630007e50 comp rx=0 tx=0).stop 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 shutdown_connections 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f662406eb70 0x7f6624071020 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6640104d60 0x7f66401153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 --2- 172.21.15.44:0/1618486277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66401043c0 0x7f6640114eb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.304+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 >> 172.21.15.44:0/1618486277 conn(0x7f66400f9fc0 msgr2=0x7f66400faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.305+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 shutdown_connections 2024-04-16T11:55:55.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.305+0000 7f6646e00640 1 -- 172.21.15.44:0/1618486277 wait complete. 2024-04-16T11:55:55.321 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-16T11:55:55.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:54 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/348980441' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-16T11:55:55.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:54 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3619317838' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-16T11:55:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 -- 172.21.15.44:0/2755983634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76501029e0 msgr2=0x7f7650064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 --2- 172.21.15.44:0/2755983634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76501029e0 0x7f7650064380 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f76380048d0 tx=0x7f7638009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 -- 172.21.15.44:0/2755983634 shutdown_connections 2024-04-16T11:55:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 --2- 172.21.15.44:0/2755983634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76501029e0 0x7f7650064380 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 --2- 172.21.15.44:0/2755983634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7650102040 0x7f7650102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 -- 172.21.15.44:0/2755983634 >> 172.21.15.44:0/2755983634 conn(0x7f7650071010 msgr2=0x7f7650071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 -- 172.21.15.44:0/2755983634 shutdown_connections 2024-04-16T11:55:55.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.526+0000 7f764fe00640 1 -- 172.21.15.44:0/2755983634 wait complete. 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.527+0000 7f764fe00640 1 Processor -- start 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.527+0000 7f764fe00640 1 -- start start 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.527+0000 7f764fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 0x7f7650114dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.527+0000 7f764fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76501029e0 0x7f7650115310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764fe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7650115910 con 0x7f7650102040 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765010ef60 con 0x7f76501029e0 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 0x7f7650114dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:55.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 0x7f7650114dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54088/0 (socket says 172.21.15.44:54088) 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 -- 172.21.15.44:0/2478913440 learned_addr learned my addr 172.21.15.44:0/2478913440 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 -- 172.21.15.44:0/2478913440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76501029e0 msgr2=0x7f7650115310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76501029e0 0x7f7650115310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 -- 172.21.15.44:0/2478913440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638005040 con 0x7f7650102040 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.528+0000 7f764ea00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 0x7f7650114dd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f764000c830 tx=0x7f7640004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.529+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764000a850 con 0x7f7650102040 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.529+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f764000ae80 con 0x7f7650102040 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.529+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7640012ea0 con 0x7f7650102040 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.529+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765010f240 con 0x7f7650102040 2024-04-16T11:55:55.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.529+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765010f760 con 0x7f7650102040 2024-04-16T11:55:55.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.530+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76500fd7e0 con 0x7f7650102040 2024-04-16T11:55:55.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.531+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7640012400 con 0x7f7650102040 2024-04-16T11:55:55.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.532+0000 7f7646a00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f763406eb70 0x7f7634071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:55.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.532+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f764008ed20 con 0x7f7650102040 2024-04-16T11:55:55.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.532+0000 7f764e000640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f763406eb70 0x7f7634071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:55.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.533+0000 7f764e000640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f763406eb70 0x7f7634071020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7638000c00 tx=0x7f7638009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:55.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.538+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f76400578d0 con 0x7f7650102040 2024-04-16T11:55:55.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.784+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f7650109230 con 0x7f7650102040 2024-04-16T11:55:55.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.785+0000 7f7646a00640 1 -- 172.21.15.44:0/2478913440 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v37) v1 ==== 107+0+4135 (secure 0 0 0) 0x7f76400126b0 con 0x7f7650102040 2024-04-16T11:55:55.783 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:55.783 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:21.139608+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:55.783 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 20 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.787+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f763406eb70 msgr2=0x7f7634071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.787+0000 7f764fe00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f763406eb70 0x7f7634071020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7638000c00 tx=0x7f7638009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.787+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 msgr2=0x7f7650114dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.787+0000 7f764fe00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 0x7f7650114dd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f764000c830 tx=0x7f7640004430 comp rx=0 tx=0).stop 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 shutdown_connections 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f763406eb70 0x7f7634071020 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76501029e0 0x7f7650115310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 --2- 172.21.15.44:0/2478913440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7650102040 0x7f7650114dd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 >> 172.21.15.44:0/2478913440 conn(0x7f7650071010 msgr2=0x7f76500fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 shutdown_connections 2024-04-16T11:55:55.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:55.788+0000 7f764fe00640 1 -- 172.21.15.44:0/2478913440 wait complete. 2024-04-16T11:55:55.804 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-16T11:55:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 -- 172.21.15.44:0/3490837155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c104d60 msgr2=0x7fc26c0fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 --2- 172.21.15.44:0/3490837155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c104d60 0x7fc26c0fe3f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc2580048d0 tx=0x7fc258009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 -- 172.21.15.44:0/3490837155 shutdown_connections 2024-04-16T11:55:56.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 --2- 172.21.15.44:0/3490837155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c104d60 0x7fc26c0fe3f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 --2- 172.21.15.44:0/3490837155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c1043c0 0x7fc26c104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 -- 172.21.15.44:0/3490837155 >> 172.21.15.44:0/3490837155 conn(0x7fc26c0f9fc0 msgr2=0x7fc26c0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:56.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 -- 172.21.15.44:0/3490837155 shutdown_connections 2024-04-16T11:55:56.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.009+0000 7fc272200640 1 -- 172.21.15.44:0/3490837155 wait complete. 2024-04-16T11:55:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.010+0000 7fc272200640 1 Processor -- start 2024-04-16T11:55:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.010+0000 7fc272200640 1 -- start start 2024-04-16T11:55:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc272200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 0x7fc26c114ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc272200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c104d60 0x7fc26c115400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc272200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c115a00 con 0x7fc26c1043c0 2024-04-16T11:55:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc272200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c10f050 con 0x7fc26c104d60 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 0x7fc26c114ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 0x7fc26c114ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54106/0 (socket says 172.21.15.44:54106) 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26b400640 1 -- 172.21.15.44:0/358424221 learned_addr learned my addr 172.21.15.44:0/358424221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26aa00640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c104d60 0x7fc26c115400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26b400640 1 -- 172.21.15.44:0/358424221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c104d60 msgr2=0x7fc26c115400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26b400640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c104d60 0x7fc26c115400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.011+0000 7fc26b400640 1 -- 172.21.15.44:0/358424221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258005040 con 0x7fc26c1043c0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.012+0000 7fc26b400640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 0x7fc26c114ec0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc25c002a50 tx=0x7fc25c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.012+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25c007c30 con 0x7fc26c1043c0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.012+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc25c00f040 con 0x7fc26c1043c0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.012+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c10f330 con 0x7fc26c1043c0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.012+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25c00db50 con 0x7fc26c1043c0 2024-04-16T11:55:56.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.012+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c10f850 con 0x7fc26c1043c0 2024-04-16T11:55:56.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.013+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c0ffb30 con 0x7fc26c1043c0 2024-04-16T11:55:56.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.014+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc25c00dcf0 con 0x7fc26c1043c0 2024-04-16T11:55:56.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.015+0000 7fc263400640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc25006eb70 0x7fc250071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:56.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.015+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fc25c08e3e0 con 0x7fc26c1043c0 2024-04-16T11:55:56.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.015+0000 7fc26aa00640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc25006eb70 0x7fc250071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:56.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.016+0000 7fc26aa00640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc25006eb70 0x7fc250071020 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc258000c00 tx=0x7fc258009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.020+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc25c056f10 con 0x7fc26c1043c0 2024-04-16T11:55:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:55 smithi044 ceph-mon[93020]: pgmap v383: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:55 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1618486277' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T11:55:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:55 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2478913440' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T11:55:56.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.265+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fc26c064380 con 0x7fc26c1043c0 2024-04-16T11:55:56.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.266+0000 7fc263400640 1 -- 172.21.15.44:0/358424221 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v37) v1 ==== 107+0+4146 (secure 0 0 0) 0x7fc25c05a5d0 con 0x7fc26c1043c0 2024-04-16T11:55:56.265 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:56.265 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:21.147121+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24277":{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":0,"incarnation":21,"state":"up:replay","state_seq":4,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:56.265 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc25006eb70 msgr2=0x7fc250071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc25006eb70 0x7fc250071020 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc258000c00 tx=0x7fc258009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 msgr2=0x7fc26c114ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 0x7fc26c114ec0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc25c002a50 tx=0x7fc25c004600 comp rx=0 tx=0).stop 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 shutdown_connections 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc25006eb70 0x7fc250071020 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c104d60 0x7fc26c115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 --2- 172.21.15.44:0/358424221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc26c1043c0 0x7fc26c114ec0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.269+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 >> 172.21.15.44:0/358424221 conn(0x7fc26c0f9fc0 msgr2=0x7fc26c1023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:56.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.270+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 shutdown_connections 2024-04-16T11:55:56.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.270+0000 7fc272200640 1 -- 172.21.15.44:0/358424221 wait complete. 2024-04-16T11:55:56.288 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-16T11:55:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:55 smithi045 ceph-mon[81423]: pgmap v383: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:55:56.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:55 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1618486277' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T11:55:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:55 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2478913440' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T11:55:56.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.505+0000 7f2411800640 1 -- 172.21.15.44:0/3426591663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 msgr2=0x7f240c107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.505+0000 7f2411800640 1 --2- 172.21.15.44:0/3426591663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c107970 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0048d0 tx=0x7f23fc009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 -- 172.21.15.44:0/3426591663 shutdown_connections 2024-04-16T11:55:56.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 --2- 172.21.15.44:0/3426591663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f240c102670 0x7f240c102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 --2- 172.21.15.44:0/3426591663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c107970 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 -- 172.21.15.44:0/3426591663 >> 172.21.15.44:0/3426591663 conn(0x7f240c071e60 msgr2=0x7f240c074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:56.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 -- 172.21.15.44:0/3426591663 shutdown_connections 2024-04-16T11:55:56.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 -- 172.21.15.44:0/3426591663 wait complete. 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.506+0000 7f2411800640 1 Processor -- start 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f2411800640 1 -- start start 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f2411800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f240c102670 0x7f240c119150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f2411800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c119690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f2411800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c119c90 con 0x7f240c1075a0 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f2411800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c1132e0 con 0x7f240c102670 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f240b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c119690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f240b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c119690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54130/0 (socket says 172.21.15.44:54130) 2024-04-16T11:55:56.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.507+0000 7f240b400640 1 -- 172.21.15.44:0/2711611421 learned_addr learned my addr 172.21.15.44:0/2711611421 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f240be00640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f240c102670 0x7f240c119150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f240b400640 1 -- 172.21.15.44:0/2711611421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f240c102670 msgr2=0x7f240c119150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f240b400640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f240c102670 0x7f240c119150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f240b400640 1 -- 172.21.15.44:0/2711611421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc005040 con 0x7f240c1075a0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f240b400640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c119690 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f23f800c810 tx=0x7f23f80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23f800a820 con 0x7f240c1075a0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23f800ae50 con 0x7f240c1075a0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c1135c0 con 0x7f240c1075a0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.508+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23f80115c0 con 0x7f240c1075a0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.509+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c113ae0 con 0x7f240c1075a0 2024-04-16T11:55:56.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.510+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c103d60 con 0x7f240c1075a0 2024-04-16T11:55:56.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.511+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23f8012420 con 0x7f240c1075a0 2024-04-16T11:55:56.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.512+0000 7f2403e00640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23f006eb70 0x7f23f0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:56.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.512+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f23f808f100 con 0x7f240c1075a0 2024-04-16T11:55:56.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.512+0000 7f240be00640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23f006eb70 0x7f23f0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:56.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.513+0000 7f240be00640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23f006eb70 0x7f23f0071020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23fc02be70 tx=0x7f23fc009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:56.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.517+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f23f8057bb0 con 0x7f240c1075a0 2024-04-16T11:55:56.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.763+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f240c10f7f0 con 0x7f240c1075a0 2024-04-16T11:55:56.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.764+0000 7f2403e00640 1 -- 172.21.15.44:0/2711611421 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v37) v1 ==== 107+0+4151 (secure 0 0 0) 0x7f23f80126d0 con 0x7f240c1075a0 2024-04-16T11:55:56.762 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:56.762 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:21.171915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24277":{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":163,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:56.762 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 22 2024-04-16T11:55:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.766+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23f006eb70 msgr2=0x7f23f0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23f006eb70 0x7f23f0071020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23fc02be70 tx=0x7f23fc009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 msgr2=0x7f240c119690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c119690 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f23f800c810 tx=0x7f23f80043c0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 shutdown_connections 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f23f006eb70 0x7f23f0071020 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075a0 0x7f240c119690 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 --2- 172.21.15.44:0/2711611421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f240c102670 0x7f240c119150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 >> 172.21.15.44:0/2711611421 conn(0x7f240c071e60 msgr2=0x7f240c104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 shutdown_connections 2024-04-16T11:55:56.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:56.767+0000 7f2411800640 1 -- 172.21.15.44:0/2711611421 wait complete. 2024-04-16T11:55:56.783 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 -- 172.21.15.44:0/829109328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104bd0 msgr2=0x7fd8500fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 --2- 172.21.15.44:0/829109328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104bd0 0x7fd8500fe260 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd8380048d0 tx=0x7fd838009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 -- 172.21.15.44:0/829109328 shutdown_connections 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 --2- 172.21.15.44:0/829109328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104bd0 0x7fd8500fe260 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 --2- 172.21.15.44:0/829109328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd850104230 0x7fd850104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 -- 172.21.15.44:0/829109328 >> 172.21.15.44:0/829109328 conn(0x7fd8500f9fc0 msgr2=0x7fd8500fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 -- 172.21.15.44:0/829109328 shutdown_connections 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.003+0000 7fd855800640 1 -- 172.21.15.44:0/829109328 wait complete. 2024-04-16T11:55:57.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.004+0000 7fd855800640 1 Processor -- start 2024-04-16T11:55:57.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.004+0000 7fd855800640 1 -- start start 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.004+0000 7fd855800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 0x7fd850114c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.004+0000 7fd855800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd850104bd0 0x7fd8501151b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.004+0000 7fd855800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8501157b0 con 0x7fd850104230 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.004+0000 7fd855800640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd850115920 con 0x7fd850104bd0 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 0x7fd850114c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 0x7fd850114c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54154/0 (socket says 172.21.15.44:54154) 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 -- 172.21.15.44:0/851079868 learned_addr learned my addr 172.21.15.44:0/851079868 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd847400640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd850104bd0 0x7fd8501151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 -- 172.21.15.44:0/851079868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd850104bd0 msgr2=0x7fd8501151b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd850104bd0 0x7fd8501151b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 -- 172.21.15.44:0/851079868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838005040 con 0x7fd850104230 2024-04-16T11:55:57.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84fe00640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 0x7fd850114c70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd84000cae0 tx=0x7fd840007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:57.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.005+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd84000ab00 con 0x7fd850104230 2024-04-16T11:55:57.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.006+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd84000ac60 con 0x7fd850104230 2024-04-16T11:55:57.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.006+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85010efb0 con 0x7fd850104230 2024-04-16T11:55:57.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.006+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8400116f0 con 0x7fd850104230 2024-04-16T11:55:57.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.006+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85010f4d0 con 0x7fd850104230 2024-04-16T11:55:57.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.007+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8500ff9a0 con 0x7fd850104230 2024-04-16T11:55:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.008+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd840012440 con 0x7fd850104230 2024-04-16T11:55:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.009+0000 7fd84cc00640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd83406eb70 0x7fd834071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.009+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fd84008f3d0 con 0x7fd850104230 2024-04-16T11:55:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.009+0000 7fd847400640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd83406eb70 0x7fd834071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.010+0000 7fd847400640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd83406eb70 0x7fd834071020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd8380047f0 tx=0x7fd838009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:57.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.014+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd840057f00 con 0x7fd850104230 2024-04-16T11:55:57.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:56 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/358424221' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T11:55:57.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:56 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2711611421' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T11:55:57.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.268+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7fd85004bcc0 con 0x7fd850104230 2024-04-16T11:55:57.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.269+0000 7fd84cc00640 1 -- 172.21.15.44:0/851079868 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v37) v1 ==== 107+0+4148 (secure 0 0 0) 0x7fd84005b5c0 con 0x7fd850104230 2024-04-16T11:55:57.267 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:22.161594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24277":{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":164,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:57.268 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 23 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd83406eb70 msgr2=0x7fd834071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd83406eb70 0x7fd834071020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd8380047f0 tx=0x7fd838009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 msgr2=0x7fd850114c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 0x7fd850114c70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd84000cae0 tx=0x7fd840007e50 comp rx=0 tx=0).stop 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 shutdown_connections 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd83406eb70 0x7fd834071020 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd850104bd0 0x7fd8501151b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 --2- 172.21.15.44:0/851079868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd850104230 0x7fd850114c70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.272+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 >> 172.21.15.44:0/851079868 conn(0x7fd8500f9fc0 msgr2=0x7fd8500fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:57.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.273+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 shutdown_connections 2024-04-16T11:55:57.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.273+0000 7fd855800640 1 -- 172.21.15.44:0/851079868 wait complete. 2024-04-16T11:55:57.290 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-16T11:55:57.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:56 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/358424221' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T11:55:57.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:56 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2711611421' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 -- 172.21.15.44:0/2148600692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 msgr2=0x7fc9640fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 --2- 172.21.15.44:0/2148600692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc9640fe260 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc9500048d0 tx=0x7fc950009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 -- 172.21.15.44:0/2148600692 shutdown_connections 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 --2- 172.21.15.44:0/2148600692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc9640fe260 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 --2- 172.21.15.44:0/2148600692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc964104230 0x7fc964104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 -- 172.21.15.44:0/2148600692 >> 172.21.15.44:0/2148600692 conn(0x7fc9640f9fc0 msgr2=0x7fc9640fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:57.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 -- 172.21.15.44:0/2148600692 shutdown_connections 2024-04-16T11:55:57.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.498+0000 7fc96a600640 1 -- 172.21.15.44:0/2148600692 wait complete. 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.499+0000 7fc96a600640 1 Processor -- start 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.499+0000 7fc96a600640 1 -- start start 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc96a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc964104230 0x7fc964114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc96a600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc964115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc96a600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc964115860 con 0x7fc964104bd0 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc96a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96410eeb0 con 0x7fc964104230 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc963e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc964115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:57.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc963e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc964115260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54172/0 (socket says 172.21.15.44:54172) 2024-04-16T11:55:57.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc963e00640 1 -- 172.21.15.44:0/3368664923 learned_addr learned my addr 172.21.15.44:0/3368664923 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:57.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc969200640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc964104230 0x7fc964114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:57.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc963e00640 1 -- 172.21.15.44:0/3368664923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc964104230 msgr2=0x7fc964114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc963e00640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc964104230 0x7fc964114d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.500+0000 7fc963e00640 1 -- 172.21.15.44:0/3368664923 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc950005040 con 0x7fc964104bd0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.501+0000 7fc963e00640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc964115260 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc9500047f0 tx=0x7fc950009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.501+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9500094c0 con 0x7fc964104bd0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.501+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc950032e50 con 0x7fc964104bd0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.501+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96410f130 con 0x7fc964104bd0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.501+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9500433b0 con 0x7fc964104bd0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.501+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96410f5f0 con 0x7fc964104bd0 2024-04-16T11:55:57.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.502+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9640ff9a0 con 0x7fc964104bd0 2024-04-16T11:55:57.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.503+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc95002f070 con 0x7fc964104bd0 2024-04-16T11:55:57.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.505+0000 7fc960c00640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc94c06eb70 0x7fc94c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.505+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fc9500b0ac0 con 0x7fc964104bd0 2024-04-16T11:55:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.505+0000 7fc969200640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc94c06eb70 0x7fc94c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:57.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.506+0000 7fc969200640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc94c06eb70 0x7fc94c071020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc958007fd0 tx=0x7fc958009e90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:57.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.509+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc950079570 con 0x7fc964104bd0 2024-04-16T11:55:57.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.757+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fc96410b390 con 0x7fc964104bd0 2024-04-16T11:55:57.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.758+0000 7fc960c00640 1 -- 172.21.15.44:0/3368664923 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v37) v1 ==== 107+0+4157 (secure 0 0 0) 0x7fc95007cc30 con 0x7fc964104bd0 2024-04-16T11:55:57.756 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:57.756 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:24.169880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24277":{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":0,"incarnation":21,"state":"up:active","state_seq":165,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24277,"qdb_cluster":[24277]},"id":1}]} 2024-04-16T11:55:57.756 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 24 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc94c06eb70 msgr2=0x7fc94c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc94c06eb70 0x7fc94c071020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc958007fd0 tx=0x7fc958009e90 comp rx=0 tx=0).stop 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 msgr2=0x7fc964115260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc964115260 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc9500047f0 tx=0x7fc950009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 shutdown_connections 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fc94c06eb70 0x7fc94c071020 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc964104bd0 0x7fc964115260 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 --2- 172.21.15.44:0/3368664923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc964104230 0x7fc964114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 >> 172.21.15.44:0/3368664923 conn(0x7fc9640f9fc0 msgr2=0x7fc9640fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 shutdown_connections 2024-04-16T11:55:57.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:57.761+0000 7fc96a600640 1 -- 172.21.15.44:0/3368664923 wait complete. 2024-04-16T11:55:57.775 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 -- 172.21.15.44:0/2470071985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe006f810 msgr2=0x7fcfe006fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/2470071985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe006f810 0x7fcfe006fbe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40048d0 tx=0x7fcfd4009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 -- 172.21.15.44:0/2470071985 shutdown_connections 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/2470071985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfe0070120 0x7fcfe01083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/2470071985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe006f810 0x7fcfe006fbe0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 -- 172.21.15.44:0/2470071985 >> 172.21.15.44:0/2470071985 conn(0x7fcfe00f7b80 msgr2=0x7fcfe00f9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 -- 172.21.15.44:0/2470071985 shutdown_connections 2024-04-16T11:55:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.002+0000 7fcfe6e00640 1 -- 172.21.15.44:0/2470071985 wait complete. 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.003+0000 7fcfe6e00640 1 Processor -- start 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.003+0000 7fcfe6e00640 1 -- start start 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.003+0000 7fcfe6e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfe006f810 0x7fcfe0114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.003+0000 7fcfe6e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 0x7fcfe01152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.003+0000 7fcfe6e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe01158c0 con 0x7fcfe0070120 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.003+0000 7fcfe6e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe010ef10 con 0x7fcfe006f810 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 0x7fcfe01152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 0x7fcfe01152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54184/0 (socket says 172.21.15.44:54184) 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5000640 1 -- 172.21.15.44:0/1630351848 learned_addr learned my addr 172.21.15.44:0/1630351848 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5a00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfe006f810 0x7fcfe0114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5000640 1 -- 172.21.15.44:0/1630351848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfe006f810 msgr2=0x7fcfe0114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5000640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfe006f810 0x7fcfe0114d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.004+0000 7fcfe5000640 1 -- 172.21.15.44:0/1630351848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4005040 con 0x7fcfe0070120 2024-04-16T11:55:58.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.005+0000 7fcfe5000640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 0x7fcfe01152c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc002a50 tx=0x7fcfcc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:58.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.005+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc007c30 con 0x7fcfe0070120 2024-04-16T11:55:58.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.005+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfcc00f040 con 0x7fcfe0070120 2024-04-16T11:55:58.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.005+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc00db50 con 0x7fcfe0070120 2024-04-16T11:55:58.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.005+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe010f1f0 con 0x7fcfe0070120 2024-04-16T11:55:58.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.005+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe010f6c0 con 0x7fcfe0070120 2024-04-16T11:55:58.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.006+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe0070f50 con 0x7fcfe0070120 2024-04-16T11:55:58.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.008+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcfcc01c400 con 0x7fcfe0070120 2024-04-16T11:55:58.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.009+0000 7fcfdd600640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcfc806eb70 0x7fcfc8071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:58.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.009+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fcfcc08e490 con 0x7fcfe0070120 2024-04-16T11:55:58.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.010+0000 7fcfe5a00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcfc806eb70 0x7fcfc8071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:58.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.010+0000 7fcfe5a00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcfc806eb70 0x7fcfc8071020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4031870 tx=0x7fcfd4009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:58.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.014+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcfcc056f40 con 0x7fcfe0070120 2024-04-16T11:55:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:57 smithi044 ceph-mon[93020]: pgmap v384: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:57 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/851079868' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T11:55:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:57 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3368664923' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T11:55:58.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.257+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fcfe004bcc0 con 0x7fcfe0070120 2024-04-16T11:55:58.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.258+0000 7fcfdd600640 1 -- 172.21.15.44:0/1630351848 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v37) v1 ==== 107+0+5004 (secure 0 0 0) 0x7fcfcc05a600 con 0x7fcfe0070120 2024-04-16T11:55:58.256 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:58.256 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:24.169880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24277},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24277":{"gid":24277,"name":"cephfs.smithi045.noofsd","rank":0,"incarnation":21,"state":"up:active","state_seq":165,"addr":"172.21.15.45:6825/4128280511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4128280511},{"type":"v1","addr":"172.21.15.45:6825","nonce":4128280511}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24277,"qdb_cluster":[24277]},"id":1}]} 2024-04-16T11:55:58.256 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-16T11:55:58.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.261+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcfc806eb70 msgr2=0x7fcfc8071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.261+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcfc806eb70 0x7fcfc8071020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4031870 tx=0x7fcfd4009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.261+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 msgr2=0x7fcfe01152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.261+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 0x7fcfe01152c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc002a50 tx=0x7fcfcc004600 comp rx=0 tx=0).stop 2024-04-16T11:55:58.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 shutdown_connections 2024-04-16T11:55:58.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcfc806eb70 0x7fcfc8071020 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0070120 0x7fcfe01152c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 --2- 172.21.15.44:0/1630351848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfe006f810 0x7fcfe0114d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 >> 172.21.15.44:0/1630351848 conn(0x7fcfe00f7b80 msgr2=0x7fcfe00f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:58.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 shutdown_connections 2024-04-16T11:55:58.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.262+0000 7fcfe6e00640 1 -- 172.21.15.44:0/1630351848 wait complete. 2024-04-16T11:55:58.279 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-16T11:55:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:57 smithi045 ceph-mon[81423]: pgmap v384: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:55:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:57 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/851079868' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T11:55:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:57 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3368664923' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T11:55:58.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.501+0000 7faaa8e00640 1 -- 172.21.15.44:0/1496211927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa4102670 msgr2=0x7faaa4102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.501+0000 7faaa8e00640 1 --2- 172.21.15.44:0/1496211927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa4102670 0x7faaa4102ae0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0048d0 tx=0x7faa8c009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.502+0000 7faaa8e00640 1 -- 172.21.15.44:0/1496211927 shutdown_connections 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.502+0000 7faaa8e00640 1 --2- 172.21.15.44:0/1496211927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa4102670 0x7faaa4102ae0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.502+0000 7faaa8e00640 1 --2- 172.21.15.44:0/1496211927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa41075a0 0x7faaa4107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.502+0000 7faaa8e00640 1 -- 172.21.15.44:0/1496211927 >> 172.21.15.44:0/1496211927 conn(0x7faaa4071e60 msgr2=0x7faaa4074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.502+0000 7faaa8e00640 1 -- 172.21.15.44:0/1496211927 shutdown_connections 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.502+0000 7faaa8e00640 1 -- 172.21.15.44:0/1496211927 wait complete. 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.503+0000 7faaa8e00640 1 Processor -- start 2024-04-16T11:55:58.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.503+0000 7faaa8e00640 1 -- start start 2024-04-16T11:55:58.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.503+0000 7faaa8e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa4102670 0x7faaa4119110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:58.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.503+0000 7faaa8e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 0x7faaa4119650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:58.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.503+0000 7faaa8e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa4119c50 con 0x7faaa41075a0 2024-04-16T11:55:58.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.503+0000 7faaa8e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa41132f0 con 0x7faaa4102670 2024-04-16T11:55:58.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 0x7faaa4119650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:58.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 0x7faaa4119650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54202/0 (socket says 172.21.15.44:54202) 2024-04-16T11:55:58.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa2a00640 1 -- 172.21.15.44:0/4093255378 learned_addr learned my addr 172.21.15.44:0/4093255378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa3400640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa4102670 0x7faaa4119110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa2a00640 1 -- 172.21.15.44:0/4093255378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa4102670 msgr2=0x7faaa4119110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa2a00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa4102670 0x7faaa4119110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa2a00640 1 -- 172.21.15.44:0/4093255378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c005040 con 0x7faaa41075a0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.504+0000 7faaa3400640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa4102670 0x7faaa4119110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.505+0000 7faaa2a00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 0x7faaa4119650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0047f0 tx=0x7faa8c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.505+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8c0094c0 con 0x7faaa41075a0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.505+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa8c032e50 con 0x7faaa41075a0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.505+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8c0433b0 con 0x7faaa41075a0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.505+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa4113570 con 0x7faaa41075a0 2024-04-16T11:55:58.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.505+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa4113a30 con 0x7faaa41075a0 2024-04-16T11:55:58.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.506+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa4103d60 con 0x7faaa41075a0 2024-04-16T11:55:58.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.507+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faa8c02f070 con 0x7faaa41075a0 2024-04-16T11:55:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.508+0000 7faa9b400640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faa8806eb70 0x7faa88071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.508+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7faa8c0b09e0 con 0x7faaa41075a0 2024-04-16T11:55:58.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.509+0000 7faaa3400640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faa8806eb70 0x7faa88071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:58.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.509+0000 7faaa3400640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faa8806eb70 0x7faa88071020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faa94009fd0 tx=0x7faa94009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:58.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.513+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faa8c079490 con 0x7faaa41075a0 2024-04-16T11:55:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.761+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7faaa410f780 con 0x7faaa41075a0 2024-04-16T11:55:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.763+0000 7faa9b400640 1 -- 172.21.15.44:0/4093255378 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v37) v1 ==== 107+0+4203 (secure 0 0 0) 0x7faa8c07cb50 con 0x7faaa41075a0 2024-04-16T11:55:58.760 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:58.761 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:53.138464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 26 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faa8806eb70 msgr2=0x7faa88071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faa8806eb70 0x7faa88071020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faa94009fd0 tx=0x7faa94009e50 comp rx=0 tx=0).stop 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 msgr2=0x7faaa4119650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 0x7faaa4119650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0047f0 tx=0x7faa8c009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 shutdown_connections 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7faa8806eb70 0x7faa88071020 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faaa41075a0 0x7faaa4119650 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 --2- 172.21.15.44:0/4093255378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faaa4102670 0x7faaa4119110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 >> 172.21.15.44:0/4093255378 conn(0x7faaa4071e60 msgr2=0x7faaa4104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 shutdown_connections 2024-04-16T11:55:58.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:58.766+0000 7faaa8e00640 1 -- 172.21.15.44:0/4093255378 wait complete. 2024-04-16T11:55:58.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-16T11:55:59.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 -- 172.21.15.44:0/1730984866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104bd0 msgr2=0x7fda440fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 --2- 172.21.15.44:0/1730984866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104bd0 0x7fda440fe260 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fda340048d0 tx=0x7fda34009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 -- 172.21.15.44:0/1730984866 shutdown_connections 2024-04-16T11:55:59.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 --2- 172.21.15.44:0/1730984866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104bd0 0x7fda440fe260 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 --2- 172.21.15.44:0/1730984866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104230 0x7fda44104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 -- 172.21.15.44:0/1730984866 >> 172.21.15.44:0/1730984866 conn(0x7fda440f9fc0 msgr2=0x7fda440fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:59.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 -- 172.21.15.44:0/1730984866 shutdown_connections 2024-04-16T11:55:59.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.003+0000 7fda4a600640 1 -- 172.21.15.44:0/1730984866 wait complete. 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.004+0000 7fda4a600640 1 Processor -- start 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.004+0000 7fda4a600640 1 -- start start 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda4a600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 0x7fda44114c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda4a600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104bd0 0x7fda441151b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda4a600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda441157b0 con 0x7fda44104230 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda4a600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda44115920 con 0x7fda44104bd0 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda49200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 0x7fda44114c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda49200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 0x7fda44114c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54214/0 (socket says 172.21.15.44:54214) 2024-04-16T11:55:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda49200640 1 -- 172.21.15.44:0/1739852408 learned_addr learned my addr 172.21.15.44:0/1739852408 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda43e00640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104bd0 0x7fda441151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda49200640 1 -- 172.21.15.44:0/1739852408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104bd0 msgr2=0x7fda441151b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda49200640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104bd0 0x7fda441151b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda49200640 1 -- 172.21.15.44:0/1739852408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda34005040 con 0x7fda44104230 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.005+0000 7fda43e00640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104bd0 0x7fda441151b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.006+0000 7fda49200640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 0x7fda44114c70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda38002a50 tx=0x7fda38004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.006+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda38007c30 con 0x7fda44104230 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.006+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda3800f040 con 0x7fda44104230 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.006+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda3800dbe0 con 0x7fda44104230 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.006+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4410efb0 con 0x7fda44104230 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.006+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4410f4d0 con 0x7fda44104230 2024-04-16T11:55:59.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.007+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda440ff9a0 con 0x7fda44104230 2024-04-16T11:55:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.008+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fda3800dd40 con 0x7fda44104230 2024-04-16T11:55:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.009+0000 7fda40c00640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fda2c06eb70 0x7fda2c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.009+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fda3808ea80 con 0x7fda44104230 2024-04-16T11:55:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.010+0000 7fda43e00640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fda2c06eb70 0x7fda2c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.010+0000 7fda43e00640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fda2c06eb70 0x7fda2c071020 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fda34000c00 tx=0x7fda34009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:59.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.015+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fda380575b0 con 0x7fda44104230 2024-04-16T11:55:59.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:58 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1630351848' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T11:55:59.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:58 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/4093255378' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T11:55:59.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.253+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fda4410b370 con 0x7fda44104230 2024-04-16T11:55:59.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.255+0000 7fda40c00640 1 -- 172.21.15.44:0/1739852408 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v37) v1 ==== 107+0+4214 (secure 0 0 0) 0x7fda3805ac70 con 0x7fda44104230 2024-04-16T11:55:59.253 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:59.253 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:53.150191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":0,"incarnation":27,"state":"up:replay","state_seq":2,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:59.253 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 27 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.257+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fda2c06eb70 msgr2=0x7fda2c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.257+0000 7fda4a600640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fda2c06eb70 0x7fda2c071020 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fda34000c00 tx=0x7fda34009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 msgr2=0x7fda44114c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 0x7fda44114c70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda38002a50 tx=0x7fda38004600 comp rx=0 tx=0).stop 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 shutdown_connections 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fda2c06eb70 0x7fda2c071020 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda44104bd0 0x7fda441151b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 --2- 172.21.15.44:0/1739852408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda44104230 0x7fda44114c70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 >> 172.21.15.44:0/1739852408 conn(0x7fda440f9fc0 msgr2=0x7fda440fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 shutdown_connections 2024-04-16T11:55:59.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.258+0000 7fda4a600640 1 -- 172.21.15.44:0/1739852408 wait complete. 2024-04-16T11:55:59.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-16T11:55:59.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:58 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1630351848' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T11:55:59.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:58 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/4093255378' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 -- 172.21.15.44:0/1896383790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104bb0 msgr2=0x7fbb900fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 --2- 172.21.15.44:0/1896383790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104bb0 0x7fbb900fe240 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c0048d0 tx=0x7fbb7c009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 -- 172.21.15.44:0/1896383790 shutdown_connections 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 --2- 172.21.15.44:0/1896383790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104bb0 0x7fbb900fe240 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 --2- 172.21.15.44:0/1896383790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb90104210 0x7fbb901045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 -- 172.21.15.44:0/1896383790 >> 172.21.15.44:0/1896383790 conn(0x7fbb900f9fc0 msgr2=0x7fbb900fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:59.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 -- 172.21.15.44:0/1896383790 shutdown_connections 2024-04-16T11:55:59.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.487+0000 7fbb97000640 1 -- 172.21.15.44:0/1896383790 wait complete. 2024-04-16T11:55:59.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.488+0000 7fbb97000640 1 Processor -- start 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.488+0000 7fbb97000640 1 -- start start 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb97000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 0x7fbb90114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb97000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb90104bb0 0x7fbb90115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb97000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90115870 con 0x7fbb90104210 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb97000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9010eec0 con 0x7fbb90104bb0 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 0x7fbb90114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 0x7fbb90114d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54246/0 (socket says 172.21.15.44:54246) 2024-04-16T11:55:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95c00640 1 -- 172.21.15.44:0/198287930 learned_addr learned my addr 172.21.15.44:0/198287930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95200640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb90104bb0 0x7fbb90115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95c00640 1 -- 172.21.15.44:0/198287930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb90104bb0 msgr2=0x7fbb90115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95c00640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb90104bb0 0x7fbb90115270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.489+0000 7fbb95c00640 1 -- 172.21.15.44:0/198287930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c005040 con 0x7fbb90104210 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.490+0000 7fbb95c00640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 0x7fbb90114d30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400c830 tx=0x7fbb84004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.490+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb8400a850 con 0x7fbb90104210 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.490+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb8400ae80 con 0x7fbb90104210 2024-04-16T11:55:59.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.490+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb84012ea0 con 0x7fbb90104210 2024-04-16T11:55:59.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.490+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9010f1a0 con 0x7fbb90104210 2024-04-16T11:55:59.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.490+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9010f6c0 con 0x7fbb90104210 2024-04-16T11:55:59.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.491+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb900ff980 con 0x7fbb90104210 2024-04-16T11:55:59.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.492+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbb84012400 con 0x7fbb90104210 2024-04-16T11:55:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.493+0000 7fbb8d600640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbb7806eb70 0x7fbb78071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.493+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fbb8408ee20 con 0x7fbb90104210 2024-04-16T11:55:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.493+0000 7fbb95200640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbb7806eb70 0x7fbb78071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.494+0000 7fbb95200640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbb7806eb70 0x7fbb78071020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c000c00 tx=0x7fbb7c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:59.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.498+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbb84057950 con 0x7fbb90104210 2024-04-16T11:55:59.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.743+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fbb9004bcc0 con 0x7fbb90104210 2024-04-16T11:55:59.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.745+0000 7fbb8d600640 1 -- 172.21.15.44:0/198287930 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v37) v1 ==== 107+0+4219 (secure 0 0 0) 0x7fbb840126b0 con 0x7fbb90104210 2024-04-16T11:55:59.743 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:55:59.743 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:53.179183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":169,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:55:59.743 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 28 2024-04-16T11:55:59.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.747+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbb7806eb70 msgr2=0x7fbb78071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.747+0000 7fbb97000640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbb7806eb70 0x7fbb78071020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c000c00 tx=0x7fbb7c009da0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.747+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 msgr2=0x7fbb90114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.747+0000 7fbb97000640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 0x7fbb90114d30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400c830 tx=0x7fbb84004430 comp rx=0 tx=0).stop 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 shutdown_connections 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fbb7806eb70 0x7fbb78071020 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb90104bb0 0x7fbb90115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 --2- 172.21.15.44:0/198287930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb90104210 0x7fbb90114d30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 >> 172.21.15.44:0/198287930 conn(0x7fbb900f9fc0 msgr2=0x7fbb900fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 shutdown_connections 2024-04-16T11:55:59.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.748+0000 7fbb97000640 1 -- 172.21.15.44:0/198287930 wait complete. 2024-04-16T11:55:59.764 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-16T11:55:59.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.983+0000 7fe09f200640 1 -- 172.21.15.44:0/4066118550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a01029e0 msgr2=0x7fe0a0064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.983+0000 7fe09f200640 1 --2- 172.21.15.44:0/4066118550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a01029e0 0x7fe0a0064380 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe088009c50 tx=0x7fe088004d10 comp rx=0 tx=0).stop 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 -- 172.21.15.44:0/4066118550 shutdown_connections 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 --2- 172.21.15.44:0/4066118550 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a01029e0 0x7fe0a0064380 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 --2- 172.21.15.44:0/4066118550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0a0102040 0x7fe0a0102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 -- 172.21.15.44:0/4066118550 >> 172.21.15.44:0/4066118550 conn(0x7fe0a0071010 msgr2=0x7fe0a0071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 -- 172.21.15.44:0/4066118550 shutdown_connections 2024-04-16T11:55:59.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 -- 172.21.15.44:0/4066118550 wait complete. 2024-04-16T11:55:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.984+0000 7fe09f200640 1 Processor -- start 2024-04-16T11:55:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09f200640 1 -- start start 2024-04-16T11:55:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 0x7fe0a0114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09f200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0a01029e0 0x7fe0a01152e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a01158e0 con 0x7fe0a0102040 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09f200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a010ef80 con 0x7fe0a01029e0 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 0x7fe0a0114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 0x7fe0a0114da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54260/0 (socket says 172.21.15.44:54260) 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.985+0000 7fe09de00640 1 -- 172.21.15.44:0/2171339506 learned_addr learned my addr 172.21.15.44:0/2171339506 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09d400640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0a01029e0 0x7fe0a01152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09de00640 1 -- 172.21.15.44:0/2171339506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0a01029e0 msgr2=0x7fe0a01152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09de00640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0a01029e0 0x7fe0a01152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:55:59.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09de00640 1 -- 172.21.15.44:0/2171339506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880098b0 con 0x7fe0a0102040 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09de00640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 0x7fe0a0114da0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe090002a50 tx=0x7fe090004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe090007c30 con 0x7fe0a0102040 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe09000f040 con 0x7fe0a0102040 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a010f260 con 0x7fe0a0102040 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe09000db50 con 0x7fe0a0102040 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.986+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a010f750 con 0x7fe0a0102040 2024-04-16T11:55:59.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.988+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a00fd7e0 con 0x7fe0a0102040 2024-04-16T11:55:59.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.989+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe09001c400 con 0x7fe0a0102040 2024-04-16T11:55:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.990+0000 7fe096000640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe08006eb70 0x7fe080071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:55:59.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.990+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fe09008e3f0 con 0x7fe0a0102040 2024-04-16T11:55:59.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.990+0000 7fe09d400640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe08006eb70 0x7fe080071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:55:59.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.991+0000 7fe09d400640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe08006eb70 0x7fe080071020 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe088009c20 tx=0x7fe08802e040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:55:59.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:55:59.995+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe090056f40 con 0x7fe0a0102040 2024-04-16T11:56:00.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:59 smithi044 ceph-mon[93020]: pgmap v385: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:00.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:59 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1739852408' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T11:56:00.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:55:59 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/198287930' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T11:56:00.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.246+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7fe0a0110440 con 0x7fe0a0102040 2024-04-16T11:56:00.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.247+0000 7fe096000640 1 -- 172.21.15.44:0/2171339506 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v37) v1 ==== 107+0+4216 (secure 0 0 0) 0x7fe09005a600 con 0x7fe0a0102040 2024-04-16T11:56:00.245 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:00.246 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:54.162763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":170,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:56:00.246 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-04-16T11:56:00.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.250+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe08006eb70 msgr2=0x7fe080071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.250+0000 7fe09f200640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe08006eb70 0x7fe080071020 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe088009c20 tx=0x7fe08802e040 comp rx=0 tx=0).stop 2024-04-16T11:56:00.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.250+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 msgr2=0x7fe0a0114da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.250+0000 7fe09f200640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 0x7fe0a0114da0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe090002a50 tx=0x7fe090004600 comp rx=0 tx=0).stop 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 shutdown_connections 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fe08006eb70 0x7fe080071020 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0a01029e0 0x7fe0a01152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 --2- 172.21.15.44:0/2171339506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe0a0102040 0x7fe0a0114da0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 >> 172.21.15.44:0/2171339506 conn(0x7fe0a0071010 msgr2=0x7fe0a00fa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 shutdown_connections 2024-04-16T11:56:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.251+0000 7fe09f200640 1 -- 172.21.15.44:0/2171339506 wait complete. 2024-04-16T11:56:00.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-16T11:56:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:59 smithi045 ceph-mon[81423]: pgmap v385: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:59 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1739852408' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T11:56:00.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:55:59 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/198287930' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T11:56:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.491+0000 7fd45d200640 1 -- 172.21.15.44:0/1520416836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd458102040 msgr2=0x7fd458102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.491+0000 7fd45d200640 1 --2- 172.21.15.44:0/1520416836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd458102040 0x7fd458102410 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd4480048d0 tx=0x7fd448009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.491+0000 7fd45d200640 1 -- 172.21.15.44:0/1520416836 shutdown_connections 2024-04-16T11:56:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.491+0000 7fd45d200640 1 --2- 172.21.15.44:0/1520416836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4581029e0 0x7fd458064380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.491+0000 7fd45d200640 1 --2- 172.21.15.44:0/1520416836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd458102040 0x7fd458102410 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.492+0000 7fd45d200640 1 -- 172.21.15.44:0/1520416836 >> 172.21.15.44:0/1520416836 conn(0x7fd458071010 msgr2=0x7fd458071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.492+0000 7fd45d200640 1 -- 172.21.15.44:0/1520416836 shutdown_connections 2024-04-16T11:56:00.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.492+0000 7fd45d200640 1 -- 172.21.15.44:0/1520416836 wait complete. 2024-04-16T11:56:00.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.492+0000 7fd45d200640 1 Processor -- start 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd45d200640 1 -- start start 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd45d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd458102040 0x7fd458114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd45d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 0x7fd4581152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd45d200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4581158c0 con 0x7fd4581029e0 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd45d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45810ef10 con 0x7fd458102040 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd44ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 0x7fd4581152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd44ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 0x7fd4581152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54276/0 (socket says 172.21.15.44:54276) 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd44ea00640 1 -- 172.21.15.44:0/943094497 learned_addr learned my addr 172.21.15.44:0/943094497 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.493+0000 7fd457400640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd458102040 0x7fd458114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44ea00640 1 -- 172.21.15.44:0/943094497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd458102040 msgr2=0x7fd458114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44ea00640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd458102040 0x7fd458114d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44ea00640 1 -- 172.21.15.44:0/943094497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd448005040 con 0x7fd4581029e0 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44ea00640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 0x7fd4581152c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd44000c760 tx=0x7fd44000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44000a7a0 con 0x7fd4581029e0 2024-04-16T11:56:00.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45810f1f0 con 0x7fd4581029e0 2024-04-16T11:56:00.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd44000add0 con 0x7fd4581029e0 2024-04-16T11:56:00.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.494+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4400113a0 con 0x7fd4581029e0 2024-04-16T11:56:00.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.495+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45810f710 con 0x7fd4581029e0 2024-04-16T11:56:00.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.496+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4580fd7e0 con 0x7fd4581029e0 2024-04-16T11:56:00.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.497+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd440012400 con 0x7fd4581029e0 2024-04-16T11:56:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.498+0000 7fd44fe00640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd43c06eb70 0x7fd43c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.498+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fd44008ed00 con 0x7fd4581029e0 2024-04-16T11:56:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.498+0000 7fd457400640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd43c06eb70 0x7fd43c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:00.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.499+0000 7fd457400640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd43c06eb70 0x7fd43c071020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd4480048a0 tx=0x7fd448009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:00.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.503+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd440057830 con 0x7fd4581029e0 2024-04-16T11:56:00.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.751+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7fd4581091f0 con 0x7fd4581029e0 2024-04-16T11:56:00.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.752+0000 7fd44fe00640 1 -- 172.21.15.44:0/943094497 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v37) v1 ==== 107+0+4225 (secure 0 0 0) 0x7fd44005aef0 con 0x7fd4581029e0 2024-04-16T11:56:00.750 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:00.750 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:56.167508+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":0,"incarnation":27,"state":"up:active","state_seq":171,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24287,"qdb_cluster":[24287]},"id":1}]} 2024-04-16T11:56:00.751 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 30 2024-04-16T11:56:00.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd43c06eb70 msgr2=0x7fd43c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd43c06eb70 0x7fd43c071020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd4480048a0 tx=0x7fd448009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 msgr2=0x7fd4581152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 0x7fd4581152c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd44000c760 tx=0x7fd44000cb60 comp rx=0 tx=0).stop 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 shutdown_connections 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd43c06eb70 0x7fd43c071020 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4581029e0 0x7fd4581152c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 --2- 172.21.15.44:0/943094497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd458102040 0x7fd458114d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.755+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 >> 172.21.15.44:0/943094497 conn(0x7fd458071010 msgr2=0x7fd4580fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.756+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 shutdown_connections 2024-04-16T11:56:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.756+0000 7fd45d200640 1 -- 172.21.15.44:0/943094497 wait complete. 2024-04-16T11:56:00.773 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-16T11:56:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.997+0000 7f5030a00640 1 -- 172.21.15.44:0/37872216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c070100 msgr2=0x7f502c1083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.997+0000 7f5030a00640 1 --2- 172.21.15.44:0/37872216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c070100 0x7f502c1083c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f50180048d0 tx=0x7f5018009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.997+0000 7f5030a00640 1 -- 172.21.15.44:0/37872216 shutdown_connections 2024-04-16T11:56:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.997+0000 7f5030a00640 1 --2- 172.21.15.44:0/37872216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c070100 0x7f502c1083c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.997+0000 7f5030a00640 1 --2- 172.21.15.44:0/37872216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f502c06f7f0 0x7f502c06fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.997+0000 7f5030a00640 1 -- 172.21.15.44:0/37872216 >> 172.21.15.44:0/37872216 conn(0x7f502c0f7b80 msgr2=0x7f502c0f9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.998+0000 7f5030a00640 1 -- 172.21.15.44:0/37872216 shutdown_connections 2024-04-16T11:56:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.998+0000 7f5030a00640 1 -- 172.21.15.44:0/37872216 wait complete. 2024-04-16T11:56:00.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.998+0000 7f5030a00640 1 Processor -- start 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.998+0000 7f5030a00640 1 -- start start 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f5030a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 0x7f502c114d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f5030a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f502c070100 0x7f502c1152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f5030a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c1158a0 con 0x7f502c06f7f0 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f5030a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c10eef0 con 0x7f502c070100 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f502b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 0x7f502c114d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f502b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 0x7f502c114d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54284/0 (socket says 172.21.15.44:54284) 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f502b400640 1 -- 172.21.15.44:0/466679488 learned_addr learned my addr 172.21.15.44:0/466679488 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:00.999+0000 7f502aa00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f502c070100 0x7f502c1152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f502b400640 1 -- 172.21.15.44:0/466679488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f502c070100 msgr2=0x7f502c1152a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f502b400640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f502c070100 0x7f502c1152a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f502b400640 1 -- 172.21.15.44:0/466679488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5018005040 con 0x7f502c06f7f0 2024-04-16T11:56:00.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f502b400640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 0x7f502c114d60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f501c00db50 tx=0x7f501c004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:00.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f501c00b9d0 con 0x7f502c06f7f0 2024-04-16T11:56:00.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f501c00f040 con 0x7f502c06f7f0 2024-04-16T11:56:00.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f501c0125d0 con 0x7f502c06f7f0 2024-04-16T11:56:01.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c10f1d0 con 0x7f502c06f7f0 2024-04-16T11:56:01.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.000+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c10f6c0 con 0x7f502c06f7f0 2024-04-16T11:56:01.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.001+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c070f30 con 0x7f502c06f7f0 2024-04-16T11:56:01.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.002+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f501c013430 con 0x7f502c06f7f0 2024-04-16T11:56:01.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.003+0000 7f5023400640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f501006eb70 0x7f5010071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.004+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f501c08f120 con 0x7f502c06f7f0 2024-04-16T11:56:01.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.004+0000 7f502aa00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f501006eb70 0x7f5010071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:01.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.004+0000 7f502aa00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f501006eb70 0x7f5010071020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f502c100760 tx=0x7f5018009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:01.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.009+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f501c057bd0 con 0x7f502c06f7f0 2024-04-16T11:56:01.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:00 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2171339506' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T11:56:01.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:00 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/943094497' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T11:56:01.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.258+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f502c04bcc0 con 0x7f502c06f7f0 2024-04-16T11:56:01.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.260+0000 7f5023400640 1 -- 172.21.15.44:0/466679488 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v37) v1 ==== 107+0+5066 (secure 0 0 0) 0x7f501c05b290 con 0x7f502c06f7f0 2024-04-16T11:56:01.258 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:01.258 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44319,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6825/33255321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":33255321},{"type":"v1","addr":"172.21.15.45:6825","nonce":33255321}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:50:56.167508+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24287":{"gid":24287,"name":"cephfs.smithi045.hxlkks","rank":0,"incarnation":27,"state":"up:active","state_seq":171,"addr":"172.21.15.45:6827/1837389700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1837389700},{"type":"v1","addr":"172.21.15.45:6827","nonce":1837389700}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24287,"qdb_cluster":[24287]},"id":1}]} 2024-04-16T11:56:01.258 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f501006eb70 msgr2=0x7f5010071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f501006eb70 0x7f5010071020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f502c100760 tx=0x7f5018009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 msgr2=0x7f502c114d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 0x7f502c114d60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f501c00db50 tx=0x7f501c004ee0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 shutdown_connections 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f501006eb70 0x7f5010071020 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f502c070100 0x7f502c1152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 --2- 172.21.15.44:0/466679488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f502c06f7f0 0x7f502c114d60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 >> 172.21.15.44:0/466679488 conn(0x7f502c0f7b80 msgr2=0x7f502c0f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 shutdown_connections 2024-04-16T11:56:01.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.263+0000 7f5030a00640 1 -- 172.21.15.44:0/466679488 wait complete. 2024-04-16T11:56:01.280 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-16T11:56:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:00 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2171339506' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T11:56:01.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:00 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/943094497' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T11:56:01.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 -- 172.21.15.44:0/1892334866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 msgr2=0x7f96fc1047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 --2- 172.21.15.44:0/1892334866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc1047c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009c50 tx=0x7f96f0004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:01.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 -- 172.21.15.44:0/1892334866 shutdown_connections 2024-04-16T11:56:01.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 --2- 172.21.15.44:0/1892334866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96fc104d90 0x7f96fc0fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 --2- 172.21.15.44:0/1892334866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc1047c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 -- 172.21.15.44:0/1892334866 >> 172.21.15.44:0/1892334866 conn(0x7f96fc0f9fc0 msgr2=0x7f96fc0fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:01.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 -- 172.21.15.44:0/1892334866 shutdown_connections 2024-04-16T11:56:01.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.502+0000 7f9704000640 1 -- 172.21.15.44:0/1892334866 wait complete. 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.503+0000 7f9704000640 1 Processor -- start 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.503+0000 7f9704000640 1 -- start start 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.503+0000 7f9704000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc114f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.503+0000 7f9704000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96fc104d90 0x7f96fc1154b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.503+0000 7f9704000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc115ab0 con 0x7f96fc1043f0 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.503+0000 7f9704000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc10f100 con 0x7f96fc104d90 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.504+0000 7f9701600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc114f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.504+0000 7f9701600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc114f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41668/0 (socket says 172.21.15.44:41668) 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.504+0000 7f9701600640 1 -- 172.21.15.44:0/1842560435 learned_addr learned my addr 172.21.15.44:0/1842560435 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:01.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.504+0000 7f9701600640 1 -- 172.21.15.44:0/1842560435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96fc104d90 msgr2=0x7f96fc1154b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.504+0000 7f9701600640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96fc104d90 0x7f96fc1154b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.504+0000 7f9701600640 1 -- 172.21.15.44:0/1842560435 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00098b0 con 0x7f96fc1043f0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.505+0000 7f9701600640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc114f70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96f002c1e0 tx=0x7f96f0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.505+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96f003a070 con 0x7f96fc1043f0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.505+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96f0004610 con 0x7f96fc1043f0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.505+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc10f380 con 0x7f96fc1043f0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.505+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96f003ebe0 con 0x7f96fc1043f0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.505+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc10f8a0 con 0x7f96fc1043f0 2024-04-16T11:56:01.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.506+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc0ffb60 con 0x7f96fc1043f0 2024-04-16T11:56:01.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.507+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96f0032070 con 0x7f96fc1043f0 2024-04-16T11:56:01.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.508+0000 7f96f9600640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f96e406eb70 0x7f96e4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.508+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f96f00b0f40 con 0x7f96fc1043f0 2024-04-16T11:56:01.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.509+0000 7f9700c00640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f96e406eb70 0x7f96e4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:01.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.509+0000 7f9700c00640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f96e406eb70 0x7f96e4071020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f96fc110830 tx=0x7f96e8009c50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:01.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.513+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f96f0079be0 con 0x7f96fc1043f0 2024-04-16T11:56:01.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.755+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f96fc10b580 con 0x7f96fc1043f0 2024-04-16T11:56:01.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.756+0000 7f96f9600640 1 -- 172.21.15.44:0/1842560435 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v37) v1 ==== 107+0+4265 (secure 0 0 0) 0x7f96f007d2a0 con 0x7f96fc1043f0 2024-04-16T11:56:01.755 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:01.755 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44319,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6825/33255321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":33255321},{"type":"v1","addr":"172.21.15.45:6825","nonce":33255321}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:51:19.475000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:56:01.756 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 32 2024-04-16T11:56:01.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f96e406eb70 msgr2=0x7f96e4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f96e406eb70 0x7f96e4071020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f96fc110830 tx=0x7f96e8009c50 comp rx=0 tx=0).stop 2024-04-16T11:56:01.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 msgr2=0x7f96fc114f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc114f70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96f002c1e0 tx=0x7f96f0004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:01.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 shutdown_connections 2024-04-16T11:56:01.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f96e406eb70 0x7f96e4071020 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96fc104d90 0x7f96fc1154b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 --2- 172.21.15.44:0/1842560435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96fc1043f0 0x7f96fc114f70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 >> 172.21.15.44:0/1842560435 conn(0x7f96fc0f9fc0 msgr2=0x7f96fc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:01.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 shutdown_connections 2024-04-16T11:56:01.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.759+0000 7f9704000640 1 -- 172.21.15.44:0/1842560435 wait complete. 2024-04-16T11:56:01.777 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-16T11:56:01.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 -- 172.21.15.44:0/2555553655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc102670 msgr2=0x7f9cbc102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 --2- 172.21.15.44:0/2555553655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc102670 0x7f9cbc102ae0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40048d0 tx=0x7f9ca4009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 -- 172.21.15.44:0/2555553655 shutdown_connections 2024-04-16T11:56:01.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 --2- 172.21.15.44:0/2555553655 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc102670 0x7f9cbc102ae0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 --2- 172.21.15.44:0/2555553655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 -- 172.21.15.44:0/2555553655 >> 172.21.15.44:0/2555553655 conn(0x7f9cbc071e60 msgr2=0x7f9cbc074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:01.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 -- 172.21.15.44:0/2555553655 shutdown_connections 2024-04-16T11:56:01.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.993+0000 7f9cc1200640 1 -- 172.21.15.44:0/2555553655 wait complete. 2024-04-16T11:56:01.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.994+0000 7f9cc1200640 1 Processor -- start 2024-04-16T11:56:01.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cc1200640 1 -- start start 2024-04-16T11:56:01.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cc1200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cbc102670 0x7f9cbc1190e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cc1200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc119620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cc1200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc119c20 con 0x7f9cbc1075a0 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cc1200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc113270 con 0x7f9cbc102670 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cb2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc119620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cb2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc119620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41684/0 (socket says 172.21.15.44:41684) 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.995+0000 7f9cb2a00640 1 -- 172.21.15.44:0/728722501 learned_addr learned my addr 172.21.15.44:0/728722501 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cbb400640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cbc102670 0x7f9cbc1190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb2a00640 1 -- 172.21.15.44:0/728722501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cbc102670 msgr2=0x7f9cbc1190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb2a00640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cbc102670 0x7f9cbc1190e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb2a00640 1 -- 172.21.15.44:0/728722501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca4005040 con 0x7f9cbc1075a0 2024-04-16T11:56:01.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb2a00640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc119620 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40047f0 tx=0x7f9ca4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:01.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ca40094c0 con 0x7f9cbc1075a0 2024-04-16T11:56:01.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ca4032e50 con 0x7f9cbc1075a0 2024-04-16T11:56:01.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ca40433b0 con 0x7f9cbc1075a0 2024-04-16T11:56:01.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.996+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc1134f0 con 0x7f9cbc1075a0 2024-04-16T11:56:01.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.997+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc1139b0 con 0x7f9cbc1075a0 2024-04-16T11:56:01.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.998+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cbc103d60 con 0x7f9cbc1075a0 2024-04-16T11:56:01.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:01.999+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ca402f070 con 0x7f9cbc1075a0 2024-04-16T11:56:01.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.000+0000 7f9cb3e00640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9ca006eb70 0x7f9ca0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:01.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.000+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f9ca40b0a10 con 0x7f9cbc1075a0 2024-04-16T11:56:01.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.001+0000 7f9cbb400640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9ca006eb70 0x7f9ca0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:01.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.001+0000 7f9cbb400640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9ca006eb70 0x7f9ca0071020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc103880 tx=0x7f9cac004d00 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:02.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.005+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9ca40794c0 con 0x7f9cbc1075a0 2024-04-16T11:56:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:01 smithi044 ceph-mon[93020]: pgmap v386: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:01 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/466679488' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T11:56:02.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:01 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1842560435' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T11:56:02.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.255+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f9cbc10f7c0 con 0x7f9cbc1075a0 2024-04-16T11:56:02.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.257+0000 7f9cb3e00640 1 -- 172.21.15.44:0/728722501 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v37) v1 ==== 107+0+4344 (secure 0 0 0) 0x7f9ca407cb80 con 0x7f9cbc1075a0 2024-04-16T11:56:02.255 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:02.255 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44319,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6825/33255321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":33255321},{"type":"v1","addr":"172.21.15.45:6825","nonce":33255321}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:51:19.487978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34432":{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:56:02.255 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 33 2024-04-16T11:56:02.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9ca006eb70 msgr2=0x7f9ca0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9ca006eb70 0x7f9ca0071020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc103880 tx=0x7f9cac004d00 comp rx=0 tx=0).stop 2024-04-16T11:56:02.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 msgr2=0x7f9cbc119620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc119620 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40047f0 tx=0x7f9ca4009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 shutdown_connections 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9ca006eb70 0x7f9ca0071020 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9cbc1075a0 0x7f9cbc119620 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.260+0000 7f9cc1200640 1 --2- 172.21.15.44:0/728722501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cbc102670 0x7f9cbc1190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.261+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 >> 172.21.15.44:0/728722501 conn(0x7f9cbc071e60 msgr2=0x7f9cbc104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.261+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 shutdown_connections 2024-04-16T11:56:02.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.261+0000 7f9cc1200640 1 -- 172.21.15.44:0/728722501 wait complete. 2024-04-16T11:56:02.276 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-16T11:56:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:01 smithi045 ceph-mon[81423]: pgmap v386: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:01 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/466679488' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T11:56:02.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:01 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1842560435' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T11:56:02.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 -- 172.21.15.44:0/131180355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c072310 msgr2=0x7fa60c10cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 --2- 172.21.15.44:0/131180355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c072310 0x7fa60c10cb30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80048d0 tx=0x7fa5f8009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 -- 172.21.15.44:0/131180355 shutdown_connections 2024-04-16T11:56:02.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 --2- 172.21.15.44:0/131180355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c072310 0x7fa60c10cb30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 --2- 172.21.15.44:0/131180355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60c071a00 0x7fa60c071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 -- 172.21.15.44:0/131180355 >> 172.21.15.44:0/131180355 conn(0x7fa60c0f9fc0 msgr2=0x7fa60c0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 -- 172.21.15.44:0/131180355 shutdown_connections 2024-04-16T11:56:02.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.491+0000 7fa614400640 1 -- 172.21.15.44:0/131180355 wait complete. 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.492+0000 7fa614400640 1 Processor -- start 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.492+0000 7fa614400640 1 -- start start 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa614400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 0x7fa60c1192c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa614400640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60c072310 0x7fa60c119800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa614400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c119e00 con 0x7fa60c071a00 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa614400640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c113450 con 0x7fa60c072310 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa611a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 0x7fa60c1192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa611a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 0x7fa60c1192c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41698/0 (socket says 172.21.15.44:41698) 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa611a00640 1 -- 172.21.15.44:0/1824030989 learned_addr learned my addr 172.21.15.44:0/1824030989 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:02.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa611a00640 1 -- 172.21.15.44:0/1824030989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60c072310 msgr2=0x7fa60c119800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa611a00640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60c072310 0x7fa60c119800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.493+0000 7fa611a00640 1 -- 172.21.15.44:0/1824030989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8005040 con 0x7fa60c071a00 2024-04-16T11:56:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.494+0000 7fa611a00640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 0x7fa60c1192c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa60000cb10 tx=0x7fa600007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.494+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa600004200 con 0x7fa60c071a00 2024-04-16T11:56:02.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.494+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa60000acf0 con 0x7fa60c071a00 2024-04-16T11:56:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.494+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa600003e40 con 0x7fa60c071a00 2024-04-16T11:56:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.494+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c113730 con 0x7fa60c071a00 2024-04-16T11:56:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.494+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c113c00 con 0x7fa60c071a00 2024-04-16T11:56:02.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.496+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c073140 con 0x7fa60c071a00 2024-04-16T11:56:02.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.496+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa60000a820 con 0x7fa60c071a00 2024-04-16T11:56:02.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.497+0000 7fa609600640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f406eb70 0x7fa5f4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:02.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.497+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fa60008f1c0 con 0x7fa60c071a00 2024-04-16T11:56:02.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.498+0000 7fa611000640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f406eb70 0x7fa5f4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:02.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.499+0000 7fa611000640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f406eb70 0x7fa5f4071020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa5f802bda0 tx=0x7fa5f8009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:02.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.502+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa600057cf0 con 0x7fa60c071a00 2024-04-16T11:56:02.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.748+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7fa60c064380 con 0x7fa60c071a00 2024-04-16T11:56:02.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.750+0000 7fa609600640 1 -- 172.21.15.44:0/1824030989 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v37) v1 ==== 107+0+4348 (secure 0 0 0) 0x7fa60000ae60 con 0x7fa60c071a00 2024-04-16T11:56:02.747 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:02.748 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44319,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6825/33255321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":33255321},{"type":"v1","addr":"172.21.15.45:6825","nonce":33255321}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:51:19.513069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34432":{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":0,"incarnation":33,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:56:02.748 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 34 2024-04-16T11:56:02.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f406eb70 msgr2=0x7fa5f4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f406eb70 0x7fa5f4071020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa5f802bda0 tx=0x7fa5f8009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 msgr2=0x7fa60c1192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 0x7fa60c1192c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa60000cb10 tx=0x7fa600007ee0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 shutdown_connections 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa5f406eb70 0x7fa5f4071020 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa60c072310 0x7fa60c119800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 --2- 172.21.15.44:0/1824030989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa60c071a00 0x7fa60c1192c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.752+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 >> 172.21.15.44:0/1824030989 conn(0x7fa60c0f9fc0 msgr2=0x7fa60c106af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.753+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 shutdown_connections 2024-04-16T11:56:02.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.753+0000 7fa614400640 1 -- 172.21.15.44:0/1824030989 wait complete. 2024-04-16T11:56:02.767 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 -- 172.21.15.44:0/3513978879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c102980 msgr2=0x7f095c0fc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 --2- 172.21.15.44:0/3513978879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c102980 0x7f095c0fc010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f09480048d0 tx=0x7f0948009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 -- 172.21.15.44:0/3513978879 shutdown_connections 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 --2- 172.21.15.44:0/3513978879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c102980 0x7f095c0fc010 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 --2- 172.21.15.44:0/3513978879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f095c101fe0 0x7f095c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 -- 172.21.15.44:0/3513978879 >> 172.21.15.44:0/3513978879 conn(0x7f095c0f7b90 msgr2=0x7f095c0f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 -- 172.21.15.44:0/3513978879 shutdown_connections 2024-04-16T11:56:02.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.981+0000 7f095be00640 1 -- 172.21.15.44:0/3513978879 wait complete. 2024-04-16T11:56:02.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.982+0000 7f095be00640 1 Processor -- start 2024-04-16T11:56:02.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.982+0000 7f095be00640 1 -- start start 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.982+0000 7f095be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 0x7f095c114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.982+0000 7f095be00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f095c102980 0x7f095c1151d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.982+0000 7f095be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1157d0 con 0x7f095c101fe0 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.982+0000 7f095be00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c10ee20 con 0x7f095c102980 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 0x7f095c114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 0x7f095c114c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41722/0 (socket says 172.21.15.44:41722) 2024-04-16T11:56:02.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 -- 172.21.15.44:0/3740225315 learned_addr learned my addr 172.21.15.44:0/3740225315 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095a000640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f095c102980 0x7f095c1151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 -- 172.21.15.44:0/3740225315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f095c102980 msgr2=0x7f095c1151d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f095c102980 0x7f095c1151d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 -- 172.21.15.44:0/3740225315 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0948005040 con 0x7f095c101fe0 2024-04-16T11:56:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.983+0000 7f095aa00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 0x7f095c114c90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f094c00db00 tx=0x7f094c004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:02.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.984+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f094c017070 con 0x7f095c101fe0 2024-04-16T11:56:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.984+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f094c00f040 con 0x7f095c101fe0 2024-04-16T11:56:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.984+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f094c003ae0 con 0x7f095c101fe0 2024-04-16T11:56:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.984+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c10f100 con 0x7f095c101fe0 2024-04-16T11:56:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.984+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c10f620 con 0x7f095c101fe0 2024-04-16T11:56:02.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.985+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c0fd750 con 0x7f095c101fe0 2024-04-16T11:56:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.986+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f094c003c40 con 0x7f095c101fe0 2024-04-16T11:56:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.987+0000 7f0952a00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f094006eb70 0x7f0940071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.987+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f094c08ecf0 con 0x7f095c101fe0 2024-04-16T11:56:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.987+0000 7f095a000640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f094006eb70 0x7f0940071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.988+0000 7f095a000640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f094006eb70 0x7f0940071020 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0948000c00 tx=0x7f0948009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:02.993+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f094c0577a0 con 0x7f095c101fe0 2024-04-16T11:56:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:02 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/728722501' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T11:56:03.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:02 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1824030989' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T11:56:03.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.241+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f095c10f840 con 0x7f095c101fe0 2024-04-16T11:56:03.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.242+0000 7f0952a00640 1 -- 172.21.15.44:0/3740225315 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v37) v1 ==== 107+0+4345 (secure 0 0 0) 0x7f094c05ae60 con 0x7f095c101fe0 2024-04-16T11:56:03.241 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:03.241 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34464,"name":"cephfs.smithi044.hlafhv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6829/2913937513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2913937513},{"type":"v1","addr":"172.21.15.44:6829","nonce":2913937513}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44319,"name":"cephfs.smithi045.noofsd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6825/33255321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":33255321},{"type":"v1","addr":"172.21.15.45:6825","nonce":33255321}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T11:39:27.511557+0000","modified":"2024-04-16T11:51:20.497626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34432":{"gid":34432,"name":"cephfs.smithi044.ufhhlc","rank":0,"incarnation":33,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.44:6827/1916629246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1916629246},{"type":"v1","addr":"172.21.15.44:6827","nonce":1916629246}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T11:56:03.241 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 35 2024-04-16T11:56:03.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f094006eb70 msgr2=0x7f0940071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:03.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f094006eb70 0x7f0940071020 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0948000c00 tx=0x7f0948009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 msgr2=0x7f095c114c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:03.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 0x7f095c114c90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f094c00db00 tx=0x7f094c004e60 comp rx=0 tx=0).stop 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 shutdown_connections 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f094006eb70 0x7f0940071020 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f095c102980 0x7f095c1151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 --2- 172.21.15.44:0/3740225315 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c101fe0 0x7f095c114c90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.245+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 >> 172.21.15.44:0/3740225315 conn(0x7f095c0f7b90 msgr2=0x7f095c070fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.246+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 shutdown_connections 2024-04-16T11:56:03.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.246+0000 7f095be00640 1 -- 172.21.15.44:0/3740225315 wait complete. 2024-04-16T11:56:03.259 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-16T11:56:03.271 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-16T11:56:03.271 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:56:03.271 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T11:56:03.325 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:56:03.325 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T11:56:03.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:02 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/728722501' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T11:56:03.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:02 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1824030989' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T11:56:03.369 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 -- 172.21.15.44:0/418328289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 msgr2=0x7fa85806f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 --2- 172.21.15.44:0/418328289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa85806f870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0048d0 tx=0x7fa84c009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 -- 172.21.15.44:0/418328289 shutdown_connections 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 --2- 172.21.15.44:0/418328289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85806fdb0 0x7fa858070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 --2- 172.21.15.44:0/418328289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa85806f870 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 -- 172.21.15.44:0/418328289 >> 172.21.15.44:0/418328289 conn(0x7fa8580f7c70 msgr2=0x7fa8580fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 -- 172.21.15.44:0/418328289 shutdown_connections 2024-04-16T11:56:03.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.630+0000 7fa85e600640 1 -- 172.21.15.44:0/418328289 wait complete. 2024-04-16T11:56:03.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.631+0000 7fa85e600640 1 Processor -- start 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.631+0000 7fa85e600640 1 -- start start 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.631+0000 7fa85e600640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85806fdb0 0x7fa8581190e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.631+0000 7fa85e600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa858119620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.631+0000 7fa85e600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858119c20 con 0x7fa8580714a0 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.631+0000 7fa85e600640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858113270 con 0x7fa85806fdb0 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa857e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa858119620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa857e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa858119620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41746/0 (socket says 172.21.15.44:41746) 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa857e00640 1 -- 172.21.15.44:0/3897930570 learned_addr learned my addr 172.21.15.44:0/3897930570 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:03.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa85d200640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85806fdb0 0x7fa8581190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa857e00640 1 -- 172.21.15.44:0/3897930570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85806fdb0 msgr2=0x7fa8581190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa857e00640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85806fdb0 0x7fa8581190e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.632+0000 7fa857e00640 1 -- 172.21.15.44:0/3897930570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c005040 con 0x7fa8580714a0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.633+0000 7fa857e00640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa858119620 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa84800c810 tx=0x7fa8480043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.633+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa84800a820 con 0x7fa8580714a0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.633+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa84800ae50 con 0x7fa8580714a0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.633+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8480115c0 con 0x7fa8580714a0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.633+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa858113550 con 0x7fa8580714a0 2024-04-16T11:56:03.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.633+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa858113a70 con 0x7fa8580714a0 2024-04-16T11:56:03.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.634+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa858070fb0 con 0x7fa8580714a0 2024-04-16T11:56:03.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.636+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa84800a980 con 0x7fa8580714a0 2024-04-16T11:56:03.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.637+0000 7fa854c00640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa84006eb80 0x7fa840071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:03.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.637+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fa84808f070 con 0x7fa8580714a0 2024-04-16T11:56:03.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.637+0000 7fa85d200640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa84006eb80 0x7fa840071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:03.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.638+0000 7fa85d200640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa84006eb80 0x7fa840071030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa84c004540 tx=0x7fa84c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:03.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.642+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa848057ba0 con 0x7fa8580714a0 2024-04-16T11:56:03.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.849+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa85810f7a0 con 0x7fa8580714a0 2024-04-16T11:56:03.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.850+0000 7fa854c00640 1 -- 172.21.15.44:0/3897930570 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+17517 (secure 0 0 0) 0x7fa84805b260 con 0x7fa8580714a0 2024-04-16T11:56:03.849 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:03.849 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:51:19.474271+0000","last_up_change":"2024-04-16T11:49:35.376417+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T11:39:26.412859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T11:39:27.033592+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":55,"up_thru":85,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6803","nonce":123479871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6805","nonce":123479871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6809","nonce":123479871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6807","nonce":123479871}]},"public_addr":"172.21.15.44:6803/123479871","cluster_addr":"172.21.15.44:6805/123479871","heartbeat_back_addr":"172.21.15.44:6809/123479871","heartbeat_front_addr":"172.21.15.44:6807/123479871","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":57,"up_from":61,"up_thru":85,"down_at":58,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6811","nonce":544437595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6813","nonce":544437595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6817","nonce":544437595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6815","nonce":544437595}]},"public_addr":"172.21.15.44:6811/544437595","cluster_addr":"172.21.15.44:6813/544437595","heartbeat_back_addr":"172.21.15.44:6817/544437595","heartbeat_front_addr":"172.21.15.44:6815/544437595","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":62,"up_from":69,"up_thru":85,"down_at":63,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6819","nonce":4137440489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6821","nonce":4137440489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6825","nonce":4137440489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6823","nonce":4137440489}]},"public_addr":"172.21.15.44:6819/4137440489","cluster_addr":"172.21.15.44:6821/4137440489","heartbeat_back_addr":"172.21.15.44:6825/4137440489","heartbeat_front_addr":"172.21.15.44:6823/4137440489","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":71,"up_from":75,"up_thru":85,"down_at":72,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6801","nonce":2814984900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6803","nonce":2814984900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6807","nonce":2814984900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6805","nonce":2814984900}]},"public_addr":"172.21.15.45:6801/2814984900","cluster_addr":"172.21.15.45:6803/2814984900","heartbeat_back_addr":"172.21.15.45:6807/2814984900","heartbeat_front_addr":"172.21.15.45:6805/2814984900","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":76,"up_from":80,"up_thru":85,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6809","nonce":3529709154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6811","nonce":3529709154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6815","nonce":3529709154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6813","nonce":3529709154}]},"public_addr":"172.21.15.45:6809/3529709154","cluster_addr":"172.21.15.45:6811/3529709154","heartbeat_back_addr":"172.21.15.45:6815/3529709154","heartbeat_front_addr":"172.21.15.45:6813/3529709154","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":81,"up_from":85,"up_thru":85,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6817","nonce":579419871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6819","nonce":579419871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6823","nonce":579419871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6821","nonce":579419871}]},"public_addr":"172.21.15.45:6817/579419871","cluster_addr":"172.21.15.45:6819/579419871","heartbeat_back_addr":"172.21.15.45:6823/579419871","heartbeat_front_addr":"172.21.15.45:6821/579419871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":57},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":62},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":71},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":76},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":81}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":5}]},{"pgid":"2.1","mappings":[{"from":0,"to":5}]},{"pgid":"2.8","mappings":[{"from":3,"to":2},{"from":0,"to":1}]},{"pgid":"2.12","mappings":[{"from":3,"to":2}]},{"pgid":"2.1a","mappings":[{"from":3,"to":2}]},{"pgid":"3.1","mappings":[{"from":0,"to":5}]},{"pgid":"3.d","mappings":[{"from":3,"to":0}]},{"pgid":"3.10","mappings":[{"from":1,"to":2}]},{"pgid":"3.11","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":3,"to":0}]},{"pgid":"3.1c","mappings":[{"from":4,"to":0},{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6827/1837389700":"2024-04-17T11:51:19.474191+0000","172.21.15.45:6826/1837389700":"2024-04-17T11:51:19.474191+0000","172.21.15.45:6825/4128280511":"2024-04-17T11:50:53.137847+0000","172.21.15.45:6824/4128280511":"2024-04-17T11:50:53.137847+0000","172.21.15.44:6829/248184983":"2024-04-17T11:50:21.139145+0000","172.21.15.44:6826/2456403196":"2024-04-17T11:49:47.142427+0000","172.21.15.44:6828/248184983":"2024-04-17T11:50:21.139145+0000","172.21.15.44:0/300728059":"2024-04-17T11:44:40.116622+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6800/2086828317":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3182081166":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/1982927697":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/3788212463":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/3480063073":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3840344332":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/1118320507":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/1480410906":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/1916760620":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6801/2086828317":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.45:6828/1075678648":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/4136480716":"2024-04-17T11:44:40.116622+0000","172.21.15.45:6829/1075678648":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/4260319814":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/2618228669":"2024-04-17T11:43:34.195423+0000","172.21.15.44:6800/3774957652":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/3774957652":"2024-04-17T11:44:40.116622+0000","172.21.15.44:6827/2456403196":"2024-04-17T11:49:47.142427+0000","172.21.15.44:0/567559671":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3726998681":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/1779875453":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/914506740":"2024-04-17T11:44:40.116622+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:56:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.853+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa84006eb80 msgr2=0x7fa840071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.853+0000 7fa85e600640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa84006eb80 0x7fa840071030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa84c004540 tx=0x7fa84c009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.853+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 msgr2=0x7fa858119620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:03.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.853+0000 7fa85e600640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa858119620 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa84800c810 tx=0x7fa8480043c0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 shutdown_connections 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa84006eb80 0x7fa840071030 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8580714a0 0x7fa858119620 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 --2- 172.21.15.44:0/3897930570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85806fdb0 0x7fa8581190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 >> 172.21.15.44:0/3897930570 conn(0x7fa8580f7c70 msgr2=0x7fa8580f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 shutdown_connections 2024-04-16T11:56:03.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:03.854+0000 7fa85e600640 1 -- 172.21.15.44:0/3897930570 wait complete. 2024-04-16T11:56:03.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T11:36:12.538436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T11:39:26.412859+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T11:39:27.033592+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T11:56:03.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 -- 172.21.15.44:0/1706284889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 msgr2=0x7f32fc06fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 --2- 172.21.15.44:0/1706284889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc06fbe0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009c50 tx=0x7f32ec004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 -- 172.21.15.44:0/1706284889 shutdown_connections 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 --2- 172.21.15.44:0/1706284889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32fc070120 0x7f32fc1083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 --2- 172.21.15.44:0/1706284889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc06fbe0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 -- 172.21.15.44:0/1706284889 >> 172.21.15.44:0/1706284889 conn(0x7f32fc0f7c70 msgr2=0x7f32fc0fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 -- 172.21.15.44:0/1706284889 shutdown_connections 2024-04-16T11:56:04.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.098+0000 7f32fb200640 1 -- 172.21.15.44:0/1706284889 wait complete. 2024-04-16T11:56:04.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.099+0000 7f32fb200640 1 Processor -- start 2024-04-16T11:56:04.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.099+0000 7f32fb200640 1 -- start start 2024-04-16T11:56:04.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.099+0000 7f32fb200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:04.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.099+0000 7f32fb200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32fc070120 0x7f32fc1152e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:04.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.099+0000 7f32fb200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc1158e0 con 0x7f32fc06f810 2024-04-16T11:56:04.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.099+0000 7f32fb200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc115a50 con 0x7f32fc070120 2024-04-16T11:56:04.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f9e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:04.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f9e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc114da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41758/0 (socket says 172.21.15.44:41758) 2024-04-16T11:56:04.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f9e00640 1 -- 172.21.15.44:0/2344877091 learned_addr learned my addr 172.21.15.44:0/2344877091 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f1400640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32fc070120 0x7f32fc1152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f9e00640 1 -- 172.21.15.44:0/2344877091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32fc070120 msgr2=0x7f32fc1152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f9e00640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32fc070120 0x7f32fc1152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.100+0000 7f32f9e00640 1 -- 172.21.15.44:0/2344877091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec0098b0 con 0x7f32fc06f810 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.101+0000 7f32f9e00640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc114da0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f32ec02c1f0 tx=0x7f32ec004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.101+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ec03a070 con 0x7f32fc06f810 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.101+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32ec004610 con 0x7f32fc06f810 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.101+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ec03ebe0 con 0x7f32fc06f810 2024-04-16T11:56:04.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.101+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc10f080 con 0x7f32fc06f810 2024-04-16T11:56:04.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.101+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc10f5a0 con 0x7f32fc06f810 2024-04-16T11:56:04.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.102+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc070f50 con 0x7f32fc06f810 2024-04-16T11:56:04.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.103+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32ec032070 con 0x7f32fc06f810 2024-04-16T11:56:04.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.104+0000 7f32f2a00640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32e006eb80 0x7f32e0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:04.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.104+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f32ec0b0f60 con 0x7f32fc06f810 2024-04-16T11:56:04.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.105+0000 7f32f1400640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32e006eb80 0x7f32e0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:04.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.105+0000 7f32f1400640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32e006eb80 0x7f32e0071030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f32e4007fd0 tx=0x7f32e4009e90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:04.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.109+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f32ec079b40 con 0x7f32fc06f810 2024-04-16T11:56:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:03 smithi044 ceph-mon[93020]: pgmap v387: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:03 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3740225315' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-16T11:56:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:03 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3897930570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:56:04.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.316+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32fc10f810 con 0x7f32fc06f810 2024-04-16T11:56:04.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.317+0000 7f32f2a00640 1 -- 172.21.15.44:0/2344877091 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32ec07d200 con 0x7f32fc06f810 2024-04-16T11:56:04.314 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.319+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32e006eb80 msgr2=0x7f32e0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.319+0000 7f32fb200640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32e006eb80 0x7f32e0071030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f32e4007fd0 tx=0x7f32e4009e90 comp rx=0 tx=0).stop 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 msgr2=0x7f32fc114da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc114da0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f32ec02c1f0 tx=0x7f32ec004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 shutdown_connections 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f32e006eb80 0x7f32e0071030 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32fc070120 0x7f32fc1152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 --2- 172.21.15.44:0/2344877091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f32fc06f810 0x7f32fc114da0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 >> 172.21.15.44:0/2344877091 conn(0x7f32fc0f7c70 msgr2=0x7f32fc0f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 shutdown_connections 2024-04-16T11:56:04.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.320+0000 7f32fb200640 1 -- 172.21.15.44:0/2344877091 wait complete. 2024-04-16T11:56:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:03 smithi045 ceph-mon[81423]: pgmap v387: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:03 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3740225315' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-16T11:56:04.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:03 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3897930570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:56:04.336 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 -- 172.21.15.44:0/948563553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81086a0 msgr2=0x7f0de8108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 --2- 172.21.15.44:0/948563553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81086a0 0x7f0de8108a70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80048d0 tx=0x7f0dd8009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 -- 172.21.15.44:0/948563553 shutdown_connections 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 --2- 172.21.15.44:0/948563553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0de81026d0 0x7f0de8102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 --2- 172.21.15.44:0/948563553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81086a0 0x7f0de8108a70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 -- 172.21.15.44:0/948563553 >> 172.21.15.44:0/948563553 conn(0x7f0de8071e70 msgr2=0x7f0de8074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 -- 172.21.15.44:0/948563553 shutdown_connections 2024-04-16T11:56:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.556+0000 7f0de7e00640 1 -- 172.21.15.44:0/948563553 wait complete. 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.557+0000 7f0de7e00640 1 Processor -- start 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.557+0000 7f0de7e00640 1 -- start start 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.557+0000 7f0de7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 0x7f0de8119130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.557+0000 7f0de7e00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0de81086a0 0x7f0de8119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.557+0000 7f0de7e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de8119c70 con 0x7f0de81026d0 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.557+0000 7f0de7e00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de81132c0 con 0x7f0de81086a0 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 0x7f0de8119130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 0x7f0de8119130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41774/0 (socket says 172.21.15.44:41774) 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 -- 172.21.15.44:0/3860731551 learned_addr learned my addr 172.21.15.44:0/3860731551 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 -- 172.21.15.44:0/3860731551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0de81086a0 msgr2=0x7f0de8119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0de81086a0 0x7f0de8119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 -- 172.21.15.44:0/3860731551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8005040 con 0x7f0de81026d0 2024-04-16T11:56:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.558+0000 7f0de6a00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 0x7f0de8119130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8004930 tx=0x7f0dd8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:04.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.559+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dd80094c0 con 0x7f0de81026d0 2024-04-16T11:56:04.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.559+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dd8032e50 con 0x7f0de81026d0 2024-04-16T11:56:04.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.559+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dd80433b0 con 0x7f0de81026d0 2024-04-16T11:56:04.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.559+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de8113540 con 0x7f0de81026d0 2024-04-16T11:56:04.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.559+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de8113a00 con 0x7f0de81026d0 2024-04-16T11:56:04.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.560+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de8103dc0 con 0x7f0de81026d0 2024-04-16T11:56:04.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.561+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0dd802f070 con 0x7f0de81026d0 2024-04-16T11:56:04.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.562+0000 7f0ddea00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0dcc06eb70 0x7f0dcc071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:04.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.562+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f0dd80b0b20 con 0x7f0de81026d0 2024-04-16T11:56:04.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.562+0000 7f0de6000640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0dcc06eb70 0x7f0dcc071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:04.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.563+0000 7f0de6000640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0dcc06eb70 0x7f0dcc071020 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0de81149f0 tx=0x7f0dd4007c30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:04.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.567+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0dd80795d0 con 0x7f0de81026d0 2024-04-16T11:56:04.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.772+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f0de810f780 con 0x7f0de81026d0 2024-04-16T11:56:04.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.773+0000 7f0ddea00640 1 -- 172.21.15.44:0/3860731551 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v91) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0dd807cc90 con 0x7f0de81026d0 2024-04-16T11:56:04.770 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0dcc06eb70 msgr2=0x7f0dcc071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0dcc06eb70 0x7f0dcc071020 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0de81149f0 tx=0x7f0dd4007c30 comp rx=0 tx=0).stop 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 msgr2=0x7f0de8119130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 0x7f0de8119130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8004930 tx=0x7f0dd8009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 shutdown_connections 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f0dcc06eb70 0x7f0dcc071020 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0de81086a0 0x7f0de8119670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 --2- 172.21.15.44:0/3860731551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0de81026d0 0x7f0de8119130 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 >> 172.21.15.44:0/3860731551 conn(0x7f0de8071e70 msgr2=0x7f0de8073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 shutdown_connections 2024-04-16T11:56:04.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:04.776+0000 7f0de7e00640 1 -- 172.21.15.44:0/3860731551 wait complete. 2024-04-16T11:56:04.790 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 -- 172.21.15.44:0/4095704057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 msgr2=0x7f68ac1027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 --2- 172.21.15.44:0/4095704057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac1027c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f689c009c50 tx=0x7f689c004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 -- 172.21.15.44:0/4095704057 shutdown_connections 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 --2- 172.21.15.44:0/4095704057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 0x7f68ac0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 --2- 172.21.15.44:0/4095704057 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac1027c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 -- 172.21.15.44:0/4095704057 >> 172.21.15.44:0/4095704057 conn(0x7f68ac0f7fe0 msgr2=0x7f68ac0fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 -- 172.21.15.44:0/4095704057 shutdown_connections 2024-04-16T11:56:05.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.024+0000 7f68b0c00640 1 -- 172.21.15.44:0/4095704057 wait complete. 2024-04-16T11:56:05.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.025+0000 7f68b0c00640 1 Processor -- start 2024-04-16T11:56:05.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.025+0000 7f68b0c00640 1 -- start start 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68b0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac115100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68b0c00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 0x7f68ac115640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68b0c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac115c40 con 0x7f68ac1023f0 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68b0c00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac10f290 con 0x7f68ac102d90 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac115100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac115100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41798/0 (socket says 172.21.15.44:41798) 2024-04-16T11:56:05.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.026+0000 7f68a9e00640 1 -- 172.21.15.44:0/581774597 learned_addr learned my addr 172.21.15.44:0/581774597 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:05.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a9400640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 0x7f68ac115640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:05.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a9e00640 1 -- 172.21.15.44:0/581774597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 msgr2=0x7f68ac115640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a9e00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 0x7f68ac115640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a9e00640 1 -- 172.21.15.44:0/581774597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c0098b0 con 0x7f68ac1023f0 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a9400640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 0x7f68ac115640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a9e00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac115100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f689c02c1f0 tx=0x7f689c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.027+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689c03a070 con 0x7f68ac1023f0 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.028+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f689c004610 con 0x7f68ac1023f0 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.028+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689c03ebe0 con 0x7f68ac1023f0 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.028+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac10f510 con 0x7f68ac1023f0 2024-04-16T11:56:05.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.028+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac10fab0 con 0x7f68ac1023f0 2024-04-16T11:56:05.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.029+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac0fdb60 con 0x7f68ac1023f0 2024-04-16T11:56:05.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.030+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f689c032070 con 0x7f68ac1023f0 2024-04-16T11:56:05.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.031+0000 7f68a2000640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f689006eb80 0x7f6890071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:05.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.031+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f689c0b0f60 con 0x7f68ac1023f0 2024-04-16T11:56:05.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.031+0000 7f68a9400640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f689006eb80 0x7f6890071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:05.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.032+0000 7f68a9400640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f689006eb80 0x7f6890071030 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f68ac1109c0 tx=0x7f689400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:05.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.036+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f689c079b40 con 0x7f68ac1023f0 2024-04-16T11:56:05.175 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:04 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2344877091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:56:05.176 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:04 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3860731551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:56:05.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.245+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f68ac109550 con 0x7f68ac1023f0 2024-04-16T11:56:05.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.245+0000 7f68a2000640 1 -- 172.21.15.44:0/581774597 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v91) v1 ==== 107+0+11 (secure 0 0 0) 0x7f689c07d200 con 0x7f68ac1023f0 2024-04-16T11:56:05.242 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:56:05.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f689006eb80 msgr2=0x7f6890071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f689006eb80 0x7f6890071030 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f68ac1109c0 tx=0x7f689400b040 comp rx=0 tx=0).stop 2024-04-16T11:56:05.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 msgr2=0x7f68ac115100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac115100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f689c02c1f0 tx=0x7f689c004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:05.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 shutdown_connections 2024-04-16T11:56:05.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f689006eb80 0x7f6890071030 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68ac102d90 0x7f68ac115640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 --2- 172.21.15.44:0/581774597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68ac1023f0 0x7f68ac115100 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 >> 172.21.15.44:0/581774597 conn(0x7f68ac0f7fe0 msgr2=0x7f68ac071150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:05.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 shutdown_connections 2024-04-16T11:56:05.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.248+0000 7f68b0c00640 1 -- 172.21.15.44:0/581774597 wait complete. 2024-04-16T11:56:05.265 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T11:56:05.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:04 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2344877091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:56:05.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:04 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3860731551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:56:05.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 -- 172.21.15.44:0/3975097415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 msgr2=0x7fcc880fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 --2- 172.21.15.44:0/3975097415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc880fe280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcc700048d0 tx=0x7fcc70009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 -- 172.21.15.44:0/3975097415 shutdown_connections 2024-04-16T11:56:05.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 --2- 172.21.15.44:0/3975097415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc880fe280 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 --2- 172.21.15.44:0/3975097415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc88104250 0x7fcc88104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 -- 172.21.15.44:0/3975097415 >> 172.21.15.44:0/3975097415 conn(0x7fcc880f9fc0 msgr2=0x7fcc880fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:05.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 -- 172.21.15.44:0/3975097415 shutdown_connections 2024-04-16T11:56:05.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.477+0000 7fcc8d200640 1 -- 172.21.15.44:0/3975097415 wait complete. 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc8d200640 1 Processor -- start 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc8d200640 1 -- start start 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc8d200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc88104250 0x7fcc88114d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc8d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc881152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc8d200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881158a0 con 0x7fcc88104bf0 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc8d200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8810eef0 con 0x7fcc88104250 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc7fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc881152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:05.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc7fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc881152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41808/0 (socket says 172.21.15.44:41808) 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.478+0000 7fcc7fe00640 1 -- 172.21.15.44:0/2949280067 learned_addr learned my addr 172.21.15.44:0/2949280067 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.479+0000 7fcc87400640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc88104250 0x7fcc88114d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.479+0000 7fcc7fe00640 1 -- 172.21.15.44:0/2949280067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc88104250 msgr2=0x7fcc88114d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.479+0000 7fcc7fe00640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc88104250 0x7fcc88114d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.479+0000 7fcc7fe00640 1 -- 172.21.15.44:0/2949280067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc70005040 con 0x7fcc88104bf0 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.479+0000 7fcc7fe00640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc881152a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc700047f0 tx=0x7fcc70009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.480+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc700094c0 con 0x7fcc88104bf0 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.480+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc70032e50 con 0x7fcc88104bf0 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.480+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8810f170 con 0x7fcc88104bf0 2024-04-16T11:56:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.480+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc700435f0 con 0x7fcc88104bf0 2024-04-16T11:56:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.480+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8810f630 con 0x7fcc88104bf0 2024-04-16T11:56:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.481+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc880ff9c0 con 0x7fcc88104bf0 2024-04-16T11:56:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.482+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc7002f070 con 0x7fcc88104bf0 2024-04-16T11:56:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.483+0000 7fcc7f400640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcc6c06eb80 0x7fcc6c071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.483+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fcc700b09e0 con 0x7fcc88104bf0 2024-04-16T11:56:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.484+0000 7fcc87400640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcc6c06eb80 0x7fcc6c071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.484+0000 7fcc87400640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcc6c06eb80 0x7fcc6c071030 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc780041d0 tx=0x7fcc78009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:05.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.488+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcc70079490 con 0x7fcc88104bf0 2024-04-16T11:56:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.715+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcc8810b440 con 0x7fcc88104bf0 2024-04-16T11:56:05.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.715+0000 7fcc7f400640 1 -- 172.21.15.44:0/2949280067 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 45 entries v91) v1 ==== 81+0+2659 (secure 0 0 0) 0x7fcc7007cb50 con 0x7fcc88104bf0 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6827/1837389700 2024-04-17T11:51:19.474191+0000 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6826/1837389700 2024-04-17T11:51:19.474191+0000 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6825/4128280511 2024-04-17T11:50:53.137847+0000 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6824/4128280511 2024-04-17T11:50:53.137847+0000 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6829/248184983 2024-04-17T11:50:21.139145+0000 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6826/2456403196 2024-04-17T11:49:47.142427+0000 2024-04-16T11:56:05.713 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6828/248184983 2024-04-17T11:50:21.139145+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/300728059 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/627376257 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2086828317 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/3690066543 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/3182081166 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/1982927697 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3788212463 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3480063073 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.714 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/630672883 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2050770335 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/3690066543 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/3840344332 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1367310856 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1118320507 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1480410906 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1916760620 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2358899774 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:05.715 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2086828317 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/627376257 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3197247130 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2358899774 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2728668058 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/483695528 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6828/1075678648 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/4136480716 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6829/1075678648 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/4260319814 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/2618228669 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/3774957652 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.716 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2653092171 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/3774957652 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6827/2456403196 2024-04-17T11:49:47.142427+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/567559671 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/679387644 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/646847999 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/3726998681 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1779875453 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.717 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/914506740 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:05.718 INFO:teuthology.orchestra.run.smithi044.stderr:listed 45 entries 2024-04-16T11:56:05.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.718+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcc6c06eb80 msgr2=0x7fcc6c071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.718+0000 7fcc8d200640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcc6c06eb80 0x7fcc6c071030 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc780041d0 tx=0x7fcc78009040 comp rx=0 tx=0).stop 2024-04-16T11:56:05.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.718+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 msgr2=0x7fcc881152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:05.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.718+0000 7fcc8d200640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc881152a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc700047f0 tx=0x7fcc70009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 shutdown_connections 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fcc6c06eb80 0x7fcc6c071030 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc88104bf0 0x7fcc881152a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 --2- 172.21.15.44:0/2949280067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc88104250 0x7fcc88114d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 >> 172.21.15.44:0/2949280067 conn(0x7fcc880f9fc0 msgr2=0x7fcc880fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 shutdown_connections 2024-04-16T11:56:05.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:05.719+0000 7fcc8d200640 1 -- 172.21.15.44:0/2949280067 wait complete. 2024-04-16T11:56:05.735 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-16T11:56:05.735 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T11:56:05.791 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.011+0000 7f9b70a00640 1 -- 172.21.15.44:0/4265535989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c102040 msgr2=0x7f9b6c102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.011+0000 7f9b70a00640 1 --2- 172.21.15.44:0/4265535989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c102040 0x7f9b6c102410 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009c50 tx=0x7f9b5c004d10 comp rx=0 tx=0).stop 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.012+0000 7f9b70a00640 1 -- 172.21.15.44:0/4265535989 shutdown_connections 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.012+0000 7f9b70a00640 1 --2- 172.21.15.44:0/4265535989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c0fc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.012+0000 7f9b70a00640 1 --2- 172.21.15.44:0/4265535989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c102040 0x7f9b6c102410 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.012+0000 7f9b70a00640 1 -- 172.21.15.44:0/4265535989 >> 172.21.15.44:0/4265535989 conn(0x7f9b6c0f7c70 msgr2=0x7f9b6c0fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.012+0000 7f9b70a00640 1 -- 172.21.15.44:0/4265535989 shutdown_connections 2024-04-16T11:56:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.012+0000 7f9b70a00640 1 -- 172.21.15.44:0/4265535989 wait complete. 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b70a00640 1 Processor -- start 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b70a00640 1 -- start start 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b70a00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c102040 0x7f9b6c114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b70a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b70a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c115860 con 0x7f9b6c1029e0 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b70a00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c10ef00 con 0x7f9b6c102040 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b69e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.013+0000 7f9b69e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c115260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41822/0 (socket says 172.21.15.44:41822) 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b69e00640 1 -- 172.21.15.44:0/2701560487 learned_addr learned my addr 172.21.15.44:0/2701560487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b6a800640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c102040 0x7f9b6c114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b69e00640 1 -- 172.21.15.44:0/2701560487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c102040 msgr2=0x7f9b6c114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b69e00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c102040 0x7f9b6c114d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b69e00640 1 -- 172.21.15.44:0/2701560487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0098b0 con 0x7f9b6c1029e0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b69e00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c115260 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c790 tx=0x7f9b580043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.014+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b5800a820 con 0x7f9b6c1029e0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.015+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b5800ae50 con 0x7f9b6c1029e0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.015+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c10f1e0 con 0x7f9b6c1029e0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.015+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b580115e0 con 0x7f9b6c1029e0 2024-04-16T11:56:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.015+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c10f780 con 0x7f9b6c1029e0 2024-04-16T11:56:06.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.016+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c0fd7b0 con 0x7f9b6c1029e0 2024-04-16T11:56:06.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.017+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9b58012400 con 0x7f9b6c1029e0 2024-04-16T11:56:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.018+0000 7f9b62a00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9b5006eb70 0x7f9b50071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.018+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f9b5808ed70 con 0x7f9b6c1029e0 2024-04-16T11:56:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.019+0000 7f9b6a800640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9b5006eb70 0x7f9b50071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.019+0000 7f9b6a800640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9b5006eb70 0x7f9b50071020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c004a30 tx=0x7f9b5c02e040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:06.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.023+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9b580578a0 con 0x7f9b6c1029e0 2024-04-16T11:56:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:05 smithi044 ceph-mon[93020]: pgmap v388: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:05 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/581774597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:56:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:05 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2949280067' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T11:56:06.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.241+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b6c109200 con 0x7f9b6c1029e0 2024-04-16T11:56:06.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.245+0000 7f9b62a00640 1 -- 172.21.15.44:0/2701560487 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+17517 (secure 0 0 0) 0x7f9b58057030 con 0x7f9b6c1029e0 2024-04-16T11:56:06.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-16T11:56:06.243 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"8b0307a8-fbe4-11ee-bc8f-c7b262605968","created":"2024-04-16T11:30:23.625043+0000","modified":"2024-04-16T11:51:19.474271+0000","last_up_change":"2024-04-16T11:49:35.376417+0000","last_in_change":"2024-04-16T11:37:26.633459+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T11:36:12.538436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T11:39:26.412859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T11:39:27.033592+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f85ef5f-daca-46b0-a83b-2fab671b920a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":55,"up_thru":85,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6803","nonce":123479871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6805","nonce":123479871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6809","nonce":123479871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":123479871},{"type":"v1","addr":"172.21.15.44:6807","nonce":123479871}]},"public_addr":"172.21.15.44:6803/123479871","cluster_addr":"172.21.15.44:6805/123479871","heartbeat_back_addr":"172.21.15.44:6809/123479871","heartbeat_front_addr":"172.21.15.44:6807/123479871","state":["exists","up"]},{"osd":1,"uuid":"8a93e9f1-a43d-4d7a-8dfe-3a77f38790d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":57,"up_from":61,"up_thru":85,"down_at":58,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6811","nonce":544437595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6813","nonce":544437595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6817","nonce":544437595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":544437595},{"type":"v1","addr":"172.21.15.44:6815","nonce":544437595}]},"public_addr":"172.21.15.44:6811/544437595","cluster_addr":"172.21.15.44:6813/544437595","heartbeat_back_addr":"172.21.15.44:6817/544437595","heartbeat_front_addr":"172.21.15.44:6815/544437595","state":["exists","up"]},{"osd":2,"uuid":"0517235e-6140-460d-9e3c-378dddaea057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":62,"up_from":69,"up_thru":85,"down_at":63,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6819","nonce":4137440489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6821","nonce":4137440489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6825","nonce":4137440489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":4137440489},{"type":"v1","addr":"172.21.15.44:6823","nonce":4137440489}]},"public_addr":"172.21.15.44:6819/4137440489","cluster_addr":"172.21.15.44:6821/4137440489","heartbeat_back_addr":"172.21.15.44:6825/4137440489","heartbeat_front_addr":"172.21.15.44:6823/4137440489","state":["exists","up"]},{"osd":3,"uuid":"d37db698-8398-425d-ac76-25d6f4c38505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":71,"up_from":75,"up_thru":85,"down_at":72,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6801","nonce":2814984900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6803","nonce":2814984900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6807","nonce":2814984900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2814984900},{"type":"v1","addr":"172.21.15.45:6805","nonce":2814984900}]},"public_addr":"172.21.15.45:6801/2814984900","cluster_addr":"172.21.15.45:6803/2814984900","heartbeat_back_addr":"172.21.15.45:6807/2814984900","heartbeat_front_addr":"172.21.15.45:6805/2814984900","state":["exists","up"]},{"osd":4,"uuid":"d58def5c-f1a7-42d0-8e0f-95acdc12b5e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":76,"up_from":80,"up_thru":85,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6809","nonce":3529709154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6811","nonce":3529709154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6815","nonce":3529709154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3529709154},{"type":"v1","addr":"172.21.15.45:6813","nonce":3529709154}]},"public_addr":"172.21.15.45:6809/3529709154","cluster_addr":"172.21.15.45:6811/3529709154","heartbeat_back_addr":"172.21.15.45:6815/3529709154","heartbeat_front_addr":"172.21.15.45:6813/3529709154","state":["exists","up"]},{"osd":5,"uuid":"8cc55233-3bbf-40ec-b61d-71fe16d1c35d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":81,"up_from":85,"up_thru":85,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6817","nonce":579419871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6819","nonce":579419871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6823","nonce":579419871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":579419871},{"type":"v1","addr":"172.21.15.45:6821","nonce":579419871}]},"public_addr":"172.21.15.45:6817/579419871","cluster_addr":"172.21.15.45:6819/579419871","heartbeat_back_addr":"172.21.15.45:6823/579419871","heartbeat_front_addr":"172.21.15.45:6821/579419871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:34:52.655565+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:35:31.764139+0000","dead_epoch":57},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:09.949912+0000","dead_epoch":62},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:36:42.987333+0000","dead_epoch":71},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:15.603902+0000","dead_epoch":76},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T11:37:48.369671+0000","dead_epoch":81}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":5}]},{"pgid":"2.1","mappings":[{"from":0,"to":5}]},{"pgid":"2.8","mappings":[{"from":3,"to":2},{"from":0,"to":1}]},{"pgid":"2.12","mappings":[{"from":3,"to":2}]},{"pgid":"2.1a","mappings":[{"from":3,"to":2}]},{"pgid":"3.1","mappings":[{"from":0,"to":5}]},{"pgid":"3.d","mappings":[{"from":3,"to":0}]},{"pgid":"3.10","mappings":[{"from":1,"to":2}]},{"pgid":"3.11","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":3,"to":0}]},{"pgid":"3.1c","mappings":[{"from":4,"to":0},{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6827/1837389700":"2024-04-17T11:51:19.474191+0000","172.21.15.45:6826/1837389700":"2024-04-17T11:51:19.474191+0000","172.21.15.45:6825/4128280511":"2024-04-17T11:50:53.137847+0000","172.21.15.45:6824/4128280511":"2024-04-17T11:50:53.137847+0000","172.21.15.44:6829/248184983":"2024-04-17T11:50:21.139145+0000","172.21.15.44:6826/2456403196":"2024-04-17T11:49:47.142427+0000","172.21.15.44:6828/248184983":"2024-04-17T11:50:21.139145+0000","172.21.15.44:0/300728059":"2024-04-17T11:44:40.116622+0000","172.21.15.44:6801/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6800/2086828317":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6801/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3182081166":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/1982927697":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/3788212463":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/3480063073":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/630672883":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/2050770335":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/3690066543":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3840344332":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/1367310856":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/1118320507":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/1480410906":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/1916760620":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6801/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6801/2086828317":"2024-04-17T11:42:16.509097+0000","172.21.15.44:6800/627376257":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/3197247130":"2024-04-17T11:31:44.879528+0000","172.21.15.44:6800/2358899774":"2024-04-17T11:31:44.879528+0000","172.21.15.44:0/2728668058":"2024-04-17T11:33:15.828378+0000","172.21.15.44:0/483695528":"2024-04-17T11:33:15.828378+0000","172.21.15.45:6828/1075678648":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/4136480716":"2024-04-17T11:44:40.116622+0000","172.21.15.45:6829/1075678648":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/4260319814":"2024-04-17T11:43:34.195423+0000","172.21.15.45:0/2618228669":"2024-04-17T11:43:34.195423+0000","172.21.15.44:6800/3774957652":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/2653092171":"2024-04-17T11:31:03.690254+0000","172.21.15.44:6801/3774957652":"2024-04-17T11:44:40.116622+0000","172.21.15.44:6827/2456403196":"2024-04-17T11:49:47.142427+0000","172.21.15.44:0/567559671":"2024-04-17T11:42:16.509097+0000","172.21.15.44:0/679387644":"2024-04-17T11:31:03.690254+0000","172.21.15.44:0/646847999":"2024-04-17T11:33:15.828378+0000","172.21.15.45:0/3726998681":"2024-04-17T11:43:34.195423+0000","172.21.15.44:0/1779875453":"2024-04-17T11:44:40.116622+0000","172.21.15.44:0/914506740":"2024-04-17T11:44:40.116622+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.247+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9b5006eb70 msgr2=0x7f9b50071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.247+0000 7f9b70a00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9b5006eb70 0x7f9b50071020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c004a30 tx=0x7f9b5c02e040 comp rx=0 tx=0).stop 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 msgr2=0x7f9b6c115260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c115260 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c790 tx=0x7f9b580043c0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 shutdown_connections 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f9b5006eb70 0x7f9b50071020 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b6c1029e0 0x7f9b6c115260 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 --2- 172.21.15.44:0/2701560487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b6c102040 0x7f9b6c114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 >> 172.21.15.44:0/2701560487 conn(0x7f9b6c0f7c70 msgr2=0x7f9b6c0f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 shutdown_connections 2024-04-16T11:56:06.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.248+0000 7f9b70a00640 1 -- 172.21.15.44:0/2701560487 wait complete. 2024-04-16T11:56:06.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T11:36:12.538436+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T11:39:26.412859+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T11:39:27.033592+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T11:56:06.263 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T11:56:06.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:05 smithi045 ceph-mon[81423]: pgmap v388: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:05 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/581774597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:56:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:05 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2949280067' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T11:56:06.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 -- 172.21.15.44:0/3928996352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 msgr2=0x7fd050102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 --2- 172.21.15.44:0/3928996352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd050102b40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd03c0048d0 tx=0x7fd03c009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 -- 172.21.15.44:0/3928996352 shutdown_connections 2024-04-16T11:56:06.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 --2- 172.21.15.44:0/3928996352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd050102b40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 --2- 172.21.15.44:0/3928996352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0501086a0 0x7fd050108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 -- 172.21.15.44:0/3928996352 >> 172.21.15.44:0/3928996352 conn(0x7fd050071e70 msgr2=0x7fd050074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:06.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 -- 172.21.15.44:0/3928996352 shutdown_connections 2024-04-16T11:56:06.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.491+0000 7fd04fe00640 1 -- 172.21.15.44:0/3928996352 wait complete. 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.492+0000 7fd04fe00640 1 Processor -- start 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.492+0000 7fd04fe00640 1 -- start start 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.492+0000 7fd04fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd0501190c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.492+0000 7fd04fe00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0501086a0 0x7fd050119600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.492+0000 7fd04fe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050119c00 con 0x7fd0501026d0 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.492+0000 7fd04fe00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050119d70 con 0x7fd0501086a0 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd0501190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd0501190c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41834/0 (socket says 172.21.15.44:41834) 2024-04-16T11:56:06.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04ea00640 1 -- 172.21.15.44:0/406420342 learned_addr learned my addr 172.21.15.44:0/406420342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04e000640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0501086a0 0x7fd050119600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04ea00640 1 -- 172.21.15.44:0/406420342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0501086a0 msgr2=0x7fd050119600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04ea00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0501086a0 0x7fd050119600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.493+0000 7fd04ea00640 1 -- 172.21.15.44:0/406420342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c005040 con 0x7fd0501026d0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.494+0000 7fd04ea00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd0501190c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd040002a50 tx=0x7fd040004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.494+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd040007c30 con 0x7fd0501026d0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.494+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd04000f040 con 0x7fd0501026d0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.494+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd050113400 con 0x7fd0501026d0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.494+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd04000db50 con 0x7fd0501026d0 2024-04-16T11:56:06.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.494+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd050113920 con 0x7fd0501026d0 2024-04-16T11:56:06.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.495+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd050103dc0 con 0x7fd0501026d0 2024-04-16T11:56:06.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.496+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd04001c400 con 0x7fd0501026d0 2024-04-16T11:56:06.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.497+0000 7fd046a00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03406eb70 0x7fd034071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.497+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fd04008e490 con 0x7fd0501026d0 2024-04-16T11:56:06.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.498+0000 7fd04e000640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03406eb70 0x7fd034071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.498+0000 7fd04e000640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03406eb70 0x7fd034071020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd03c003080 tx=0x7fd03c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:06.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.503+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd040056f40 con 0x7fd0501026d0 2024-04-16T11:56:06.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.711+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd050113b40 con 0x7fd0501026d0 2024-04-16T11:56:06.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.712+0000 7fd046a00640 1 -- 172.21.15.44:0/406420342 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd04005a600 con 0x7fd0501026d0 2024-04-16T11:56:06.710 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.715+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03406eb70 msgr2=0x7fd034071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.715+0000 7fd04fe00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03406eb70 0x7fd034071020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd03c003080 tx=0x7fd03c009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.715+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 msgr2=0x7fd0501190c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.715+0000 7fd04fe00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd0501190c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd040002a50 tx=0x7fd040004600 comp rx=0 tx=0).stop 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 shutdown_connections 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd03406eb70 0x7fd034071020 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0501086a0 0x7fd050119600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 --2- 172.21.15.44:0/406420342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd0501026d0 0x7fd0501190c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 >> 172.21.15.44:0/406420342 conn(0x7fd050071e70 msgr2=0x7fd050073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:06.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 shutdown_connections 2024-04-16T11:56:06.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.716+0000 7fd04fe00640 1 -- 172.21.15.44:0/406420342 wait complete. 2024-04-16T11:56:06.733 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T11:56:06.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 -- 172.21.15.44:0/1340354015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104410 msgr2=0x7f40c01047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 --2- 172.21.15.44:0/1340354015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104410 0x7f40c01047e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f40b00048d0 tx=0x7f40b0009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 -- 172.21.15.44:0/1340354015 shutdown_connections 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 --2- 172.21.15.44:0/1340354015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40c0104db0 0x7f40c00fe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 --2- 172.21.15.44:0/1340354015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104410 0x7f40c01047e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 -- 172.21.15.44:0/1340354015 >> 172.21.15.44:0/1340354015 conn(0x7f40c00fa020 msgr2=0x7f40c00fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 -- 172.21.15.44:0/1340354015 shutdown_connections 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.964+0000 7f40c7000640 1 -- 172.21.15.44:0/1340354015 wait complete. 2024-04-16T11:56:06.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.965+0000 7f40c7000640 1 Processor -- start 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.965+0000 7f40c7000640 1 -- start start 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.965+0000 7f40c7000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40c0104410 0x7f40c0114fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.965+0000 7f40c7000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 0x7f40c01154e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.965+0000 7f40c7000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c0115b70 con 0x7f40c0104db0 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.965+0000 7f40c7000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c010f130 con 0x7f40c0104410 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40b7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 0x7f40c01154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40b7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 0x7f40c01154e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41850/0 (socket says 172.21.15.44:41850) 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40b7400640 1 -- 172.21.15.44:0/1524133149 learned_addr learned my addr 172.21.15.44:0/1524133149 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40bfe00640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40c0104410 0x7f40c0114fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40b7400640 1 -- 172.21.15.44:0/1524133149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40c0104410 msgr2=0x7f40c0114fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40b7400640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40c0104410 0x7f40c0114fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.966+0000 7f40b7400640 1 -- 172.21.15.44:0/1524133149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0005040 con 0x7f40c0104db0 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.967+0000 7f40b7400640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 0x7f40c01154e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f40a800b790 tx=0x7f40a80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.967+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a800c700 con 0x7f40c0104db0 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.967+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c010f410 con 0x7f40c0104db0 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.967+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40a800cd30 con 0x7f40c0104db0 2024-04-16T11:56:06.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.967+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a8013370 con 0x7f40c0104db0 2024-04-16T11:56:06.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.967+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c010f930 con 0x7f40c0104db0 2024-04-16T11:56:06.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.968+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c00ffb80 con 0x7f40c0104db0 2024-04-16T11:56:06.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.969+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f40a8012070 con 0x7f40c0104db0 2024-04-16T11:56:06.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.971+0000 7f40bcc00640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f40a406eb80 0x7f40a4071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:06.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.971+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7f40a808e540 con 0x7f40c0104db0 2024-04-16T11:56:06.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.971+0000 7f40bfe00640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f40a406eb80 0x7f40a4071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:06.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.972+0000 7f40bfe00640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f40a406eb80 0x7f40a4071030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f40b00048a0 tx=0x7f40b0009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:06.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:06.976+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f40a8056ff0 con 0x7f40c0104db0 2024-04-16T11:56:07.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.184+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f40c010b630 con 0x7f40c0104db0 2024-04-16T11:56:07.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.185+0000 7f40bcc00640 1 -- 172.21.15.44:0/1524133149 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v91) v1 ==== 107+0+11 (secure 0 0 0) 0x7f40a805a6b0 con 0x7f40c0104db0 2024-04-16T11:56:07.182 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.187+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f40a406eb80 msgr2=0x7f40a4071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.187+0000 7f40c7000640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f40a406eb80 0x7f40a4071030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f40b00048a0 tx=0x7f40b0009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.187+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 msgr2=0x7f40c01154e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.187+0000 7f40c7000640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 0x7f40c01154e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f40a800b790 tx=0x7f40a80091d0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 shutdown_connections 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7f40a406eb80 0x7f40a4071030 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f40c0104db0 0x7f40c01154e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 --2- 172.21.15.44:0/1524133149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40c0104410 0x7f40c0114fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 >> 172.21.15.44:0/1524133149 conn(0x7f40c00fa020 msgr2=0x7f40c00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:07.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 shutdown_connections 2024-04-16T11:56:07.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.188+0000 7f40c7000640 1 -- 172.21.15.44:0/1524133149 wait complete. 2024-04-16T11:56:07.201 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:06 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/2701560487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:56:07.201 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:06 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/406420342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:56:07.206 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T11:56:07.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:06 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/2701560487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T11:56:07.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:06 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/406420342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T11:56:07.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 -- 172.21.15.44:0/1458399811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fdd20 msgr2=0x7fa4bc1063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 --2- 172.21.15.44:0/1458399811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fdd20 0x7fa4bc1063f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80048d0 tx=0x7fa4a8009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 -- 172.21.15.44:0/1458399811 shutdown_connections 2024-04-16T11:56:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 --2- 172.21.15.44:0/1458399811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fdd20 0x7fa4bc1063f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 --2- 172.21.15.44:0/1458399811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc0fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 -- 172.21.15.44:0/1458399811 >> 172.21.15.44:0/1458399811 conn(0x7fa4bc0f6e80 msgr2=0x7fa4bc0f9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 -- 172.21.15.44:0/1458399811 shutdown_connections 2024-04-16T11:56:07.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.435+0000 7fa4c2200640 1 -- 172.21.15.44:0/1458399811 wait complete. 2024-04-16T11:56:07.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.436+0000 7fa4c2200640 1 Processor -- start 2024-04-16T11:56:07.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.436+0000 7fa4c2200640 1 -- start start 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.436+0000 7fa4c2200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc112f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.436+0000 7fa4c2200640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4bc0fdd20 0x7fa4bc1134d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.436+0000 7fa4c2200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc113ad0 con 0x7fa4bc0fd410 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.436+0000 7fa4c2200640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc10d120 con 0x7fa4bc0fdd20 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4bb400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc112f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4bb400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc112f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41870/0 (socket says 172.21.15.44:41870) 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4bb400640 1 -- 172.21.15.44:0/3320436280 learned_addr learned my addr 172.21.15.44:0/3320436280 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4baa00640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4bc0fdd20 0x7fa4bc1134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4bb400640 1 -- 172.21.15.44:0/3320436280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4bc0fdd20 msgr2=0x7fa4bc1134d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4bb400640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4bc0fdd20 0x7fa4bc1134d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.437+0000 7fa4bb400640 1 -- 172.21.15.44:0/3320436280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a8005040 con 0x7fa4bc0fd410 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.438+0000 7fa4bb400640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc112f90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00c810 tx=0x7fa4ac0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.438+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4ac00a820 con 0x7fa4bc0fd410 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.438+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc10d400 con 0x7fa4bc0fd410 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.438+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4ac00ae50 con 0x7fa4bc0fd410 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.438+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4ac01b5f0 con 0x7fa4bc0fd410 2024-04-16T11:56:07.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.438+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc10d920 con 0x7fa4bc0fd410 2024-04-16T11:56:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.439+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc065e40 con 0x7fa4bc0fd410 2024-04-16T11:56:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.440+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa4ac00a980 con 0x7fa4bc0fd410 2024-04-16T11:56:07.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.441+0000 7fa4b3400640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa4a006eb80 0x7fa4a0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:07.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.441+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fa4ac08f050 con 0x7fa4bc0fd410 2024-04-16T11:56:07.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.441+0000 7fa4baa00640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa4a006eb80 0x7fa4a0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:07.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.442+0000 7fa4baa00640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa4a006eb80 0x7fa4a0071030 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009fa0 tx=0x7fa4a8009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:07.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.447+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa4ac057ba0 con 0x7fa4bc0fd410 2024-04-16T11:56:07.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.654+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fa4bc109580 con 0x7fa4bc0fd410 2024-04-16T11:56:07.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.655+0000 7fa4b3400640 1 -- 172.21.15.44:0/3320436280 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v91) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa4ac05b260 con 0x7fa4bc0fd410 2024-04-16T11:56:07.652 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-16T11:56:07.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa4a006eb80 msgr2=0x7fa4a0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa4a006eb80 0x7fa4a0071030 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009fa0 tx=0x7fa4a8009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 msgr2=0x7fa4bc112f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc112f90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00c810 tx=0x7fa4ac0043c0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 shutdown_connections 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fa4a006eb80 0x7fa4a0071030 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4bc0fdd20 0x7fa4bc1134d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 --2- 172.21.15.44:0/3320436280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa4bc0fd410 0x7fa4bc112f90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 >> 172.21.15.44:0/3320436280 conn(0x7fa4bc0f6e80 msgr2=0x7fa4bc0f83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 shutdown_connections 2024-04-16T11:56:07.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.658+0000 7fa4c2200640 1 -- 172.21.15.44:0/3320436280 wait complete. 2024-04-16T11:56:07.676 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T11:56:07.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.898+0000 7fd67f000640 1 -- 172.21.15.44:0/696468674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd678102040 msgr2=0x7fd678102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.898+0000 7fd67f000640 1 --2- 172.21.15.44:0/696468674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd678102040 0x7fd678102410 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0048d0 tx=0x7fd66c009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 -- 172.21.15.44:0/696468674 shutdown_connections 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 --2- 172.21.15.44:0/696468674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6781029e0 0x7fd678064400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 --2- 172.21.15.44:0/696468674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd678102040 0x7fd678102410 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 -- 172.21.15.44:0/696468674 >> 172.21.15.44:0/696468674 conn(0x7fd678071df0 msgr2=0x7fd6780721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 -- 172.21.15.44:0/696468674 shutdown_connections 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 -- 172.21.15.44:0/696468674 wait complete. 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.899+0000 7fd67f000640 1 Processor -- start 2024-04-16T11:56:07.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67f000640 1 -- start start 2024-04-16T11:56:07.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67f000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd678102040 0x7fd678114e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67f000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 0x7fd678115360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67f000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678115960 con 0x7fd6781029e0 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67f000640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67810f000 con 0x7fd678102040 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 0x7fd678115360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 0x7fd678115360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41882/0 (socket says 172.21.15.44:41882) 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.900+0000 7fd67d200640 1 -- 172.21.15.44:0/3183510791 learned_addr learned my addr 172.21.15.44:0/3183510791 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67dc00640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd678102040 0x7fd678114e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67d200640 1 -- 172.21.15.44:0/3183510791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd678102040 msgr2=0x7fd678114e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67d200640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd678102040 0x7fd678114e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:07.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67d200640 1 -- 172.21.15.44:0/3183510791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c005040 con 0x7fd6781029e0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67d200640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 0x7fd678115360 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd664002840 tx=0x7fd664004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd664007c30 con 0x7fd6781029e0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd66400f040 con 0x7fd6781029e0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd664009590 con 0x7fd6781029e0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67810f2e0 con 0x7fd6781029e0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.901+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67810f800 con 0x7fd6781029e0 2024-04-16T11:56:07.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.902+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6780fd7c0 con 0x7fd6781029e0 2024-04-16T11:56:07.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.903+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 37) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd66400d6b0 con 0x7fd6781029e0 2024-04-16T11:56:07.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.905+0000 7fd675600640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd66006eb70 0x7fd660071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T11:56:07.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.905+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 7089+0+0 (secure 0 0 0) 0x7fd66408e320 con 0x7fd6781029e0 2024-04-16T11:56:07.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.905+0000 7fd67dc00640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd66006eb70 0x7fd660071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T11:56:07.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.906+0000 7fd67dc00640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd66006eb70 0x7fd660071020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd66c003080 tx=0x7fd66c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T11:56:07.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:07.909+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd664056e50 con 0x7fd6781029e0 2024-04-16T11:56:08.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.130+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd678109200 con 0x7fd6781029e0 2024-04-16T11:56:08.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.131+0000 7fd675600640 1 -- 172.21.15.44:0/3183510791 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 45 entries v91) v1 ==== 81+0+2659 (secure 0 0 0) 0x7fd66405a510 con 0x7fd6781029e0 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6827/1837389700 2024-04-17T11:51:19.474191+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6826/1837389700 2024-04-17T11:51:19.474191+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6825/4128280511 2024-04-17T11:50:53.137847+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6824/4128280511 2024-04-17T11:50:53.137847+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6829/248184983 2024-04-17T11:50:21.139145+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6826/2456403196 2024-04-17T11:49:47.142427+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6828/248184983 2024-04-17T11:50:21.139145+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/300728059 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/627376257 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2086828317 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/3690066543 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/3182081166 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/1982927697 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.129 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3788212463 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3480063073 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/630672883 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2050770335 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/3690066543 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/3840344332 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1367310856 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1118320507 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1480410906 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1916760620 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2358899774 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2086828317 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/627376257 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3197247130 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:08.130 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2358899774 2024-04-17T11:31:44.879528+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2728668058 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/483695528 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6828/1075678648 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/4136480716 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:6829/1075678648 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/4260319814 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/2618228669 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/3774957652 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2653092171 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/3774957652 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6827/2456403196 2024-04-17T11:49:47.142427+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/567559671 2024-04-17T11:42:16.509097+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/679387644 2024-04-17T11:31:03.690254+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/646847999 2024-04-17T11:33:15.828378+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.45:0/3726998681 2024-04-17T11:43:34.195423+0000 2024-04-16T11:56:08.131 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1779875453 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.132 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/914506740 2024-04-17T11:44:40.116622+0000 2024-04-16T11:56:08.132 INFO:teuthology.orchestra.run.smithi044.stderr:listed 45 entries 2024-04-16T11:56:08.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd66006eb70 msgr2=0x7fd660071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:08.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd66006eb70 0x7fd660071020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd66c003080 tx=0x7fd66c009da0 comp rx=0 tx=0).stop 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 msgr2=0x7fd678115360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 0x7fd678115360 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd664002840 tx=0x7fd664004570 comp rx=0 tx=0).stop 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 shutdown_connections 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:6800/2084563884,v1:172.21.15.44:6801/2084563884] conn(0x7fd66006eb70 0x7fd660071020 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6781029e0 0x7fd678115360 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 --2- 172.21.15.44:0/3183510791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd678102040 0x7fd678114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.134+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 >> 172.21.15.44:0/3183510791 conn(0x7fd678071df0 msgr2=0x7fd678102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.135+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 shutdown_connections 2024-04-16T11:56:08.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-16T11:56:08.135+0000 7fd67f000640 1 -- 172.21.15.44:0/3183510791 wait complete. 2024-04-16T11:56:08.148 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-16T11:56:08.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T11:56:08.148 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-16T11:56:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:07 smithi044 ceph-mon[93020]: pgmap v389: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:07 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/1524133149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:56:08.225 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:07 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3320436280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:56:08.249 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T11:56:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:07 smithi045 ceph-mon[81423]: pgmap v389: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:07 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/1524133149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T11:56:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:07 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3320436280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T11:56:09.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:08 smithi044 ceph-mon[93020]: from='client.? 172.21.15.44:0/3183510791' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T11:56:09.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:08 smithi045 ceph-mon[81423]: from='client.? 172.21.15.44:0/3183510791' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T11:56:10.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:09 smithi044 ceph-mon[93020]: pgmap v390: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:10.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:09 smithi045 ceph-mon[81423]: pgmap v390: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:11.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:12.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:11 smithi045 ceph-mon[81423]: pgmap v391: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:12.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:11 smithi044 ceph-mon[93020]: pgmap v391: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:14.279 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:13 smithi044 ceph-mon[93020]: pgmap v392: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:14.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:13 smithi045 ceph-mon[81423]: pgmap v392: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:16.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:15 smithi045 ceph-mon[81423]: pgmap v393: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:15 smithi044 ceph-mon[93020]: pgmap v393: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:18.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:18 smithi045 ceph-mon[81423]: pgmap v394: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:18 smithi044 ceph-mon[93020]: pgmap v394: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:20 smithi045 ceph-mon[81423]: pgmap v395: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:20 smithi044 ceph-mon[93020]: pgmap v395: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:22.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:22 smithi045 ceph-mon[81423]: pgmap v396: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:22.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:22 smithi044 ceph-mon[93020]: pgmap v396: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:24.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:24 smithi045 ceph-mon[81423]: pgmap v397: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:24 smithi044 ceph-mon[93020]: pgmap v397: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:26.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:26 smithi045 ceph-mon[81423]: pgmap v398: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:26.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:26.397 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:26 smithi044 ceph-mon[93020]: pgmap v398: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:26.397 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:28.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:28 smithi045 ceph-mon[81423]: pgmap v399: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:28 smithi044 ceph-mon[93020]: pgmap v399: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:30.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:30 smithi045 ceph-mon[81423]: pgmap v400: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:30.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:30 smithi044 ceph-mon[93020]: pgmap v400: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:32.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:32 smithi045 ceph-mon[81423]: pgmap v401: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:32.475 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:32 smithi044 ceph-mon[93020]: pgmap v401: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:33.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:33 smithi045 ceph-mon[81423]: pgmap v402: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:33.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:33 smithi044 ceph-mon[93020]: pgmap v402: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:36.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:35 smithi045 ceph-mon[81423]: pgmap v403: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:36.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:35 smithi044 ceph-mon[93020]: pgmap v403: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:38.080 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:37 smithi045 ceph-mon[81423]: pgmap v404: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:37 smithi044 ceph-mon[93020]: pgmap v404: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:40.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:39 smithi044 ceph-mon[93020]: pgmap v405: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:40.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:39 smithi045 ceph-mon[81423]: pgmap v405: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:56:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:41.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:56:41.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:41.935 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:41 smithi045 ceph-mon[81423]: pgmap v406: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:41.935 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:56:41.935 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:56:41.935 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:56:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:41 smithi044 ceph-mon[93020]: pgmap v406: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:56:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:56:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:56:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:43 smithi044 ceph-mon[93020]: pgmap v407: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:44.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:43 smithi045 ceph-mon[81423]: pgmap v407: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:46.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:45 smithi044 ceph-mon[93020]: pgmap v408: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:46.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:45 smithi045 ceph-mon[81423]: pgmap v408: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:47 smithi044 ceph-mon[93020]: pgmap v409: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:48.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:47 smithi045 ceph-mon[81423]: pgmap v409: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:49 smithi044 ceph-mon[93020]: pgmap v410: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:50.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:49 smithi045 ceph-mon[81423]: pgmap v410: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:51 smithi044 ceph-mon[93020]: pgmap v411: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:52.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:51 smithi045 ceph-mon[81423]: pgmap v411: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:53 smithi044 ceph-mon[93020]: pgmap v412: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:54.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:53 smithi045 ceph-mon[81423]: pgmap v412: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:55 smithi044 ceph-mon[93020]: pgmap v413: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:55 smithi045 ceph-mon[81423]: pgmap v413: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:56:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:57 smithi044 ceph-mon[93020]: pgmap v414: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:56:58.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:57 smithi045 ceph-mon[81423]: pgmap v414: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:56:59 smithi044 ceph-mon[93020]: pgmap v415: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:56:59 smithi045 ceph-mon[81423]: pgmap v415: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:01 smithi044 ceph-mon[93020]: pgmap v416: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:02.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:01 smithi045 ceph-mon[81423]: pgmap v416: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:03 smithi044 ceph-mon[93020]: pgmap v417: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:03 smithi045 ceph-mon[81423]: pgmap v417: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:05 smithi044 ceph-mon[93020]: pgmap v418: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:05 smithi045 ceph-mon[81423]: pgmap v418: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:07 smithi044 ceph-mon[93020]: pgmap v419: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:08.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:07 smithi045 ceph-mon[81423]: pgmap v419: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:10.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:09 smithi044 ceph-mon[93020]: pgmap v420: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:10.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:09 smithi045 ceph-mon[81423]: pgmap v420: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:11.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:12.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:11 smithi044 ceph-mon[93020]: pgmap v421: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:12.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:11 smithi045 ceph-mon[81423]: pgmap v421: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:13 smithi044 ceph-mon[93020]: pgmap v422: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:13 smithi045 ceph-mon[81423]: pgmap v422: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:15 smithi044 ceph-mon[93020]: pgmap v423: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:15 smithi045 ceph-mon[81423]: pgmap v423: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:17 smithi044 ceph-mon[93020]: pgmap v424: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:18.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:17 smithi045 ceph-mon[81423]: pgmap v424: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:19 smithi045 ceph-mon[81423]: pgmap v425: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:19 smithi044 ceph-mon[93020]: pgmap v425: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:22.329 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:21 smithi045 ceph-mon[81423]: pgmap v426: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:22.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:21 smithi044 ceph-mon[93020]: pgmap v426: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:24.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:23 smithi045 ceph-mon[81423]: pgmap v427: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:23 smithi044 ceph-mon[93020]: pgmap v427: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:26.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:26 smithi045 ceph-mon[81423]: pgmap v428: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:26.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:26.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:25 smithi044 ceph-mon[93020]: pgmap v428: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:26.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:28.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:28 smithi045 ceph-mon[81423]: pgmap v429: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:28 smithi044 ceph-mon[93020]: pgmap v429: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:30.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:30 smithi045 ceph-mon[81423]: pgmap v430: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:30 smithi044 ceph-mon[93020]: pgmap v430: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:32 smithi045 ceph-mon[81423]: pgmap v431: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:32 smithi044 ceph-mon[93020]: pgmap v431: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:34 smithi045 ceph-mon[81423]: pgmap v432: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:34.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:34 smithi044 ceph-mon[93020]: pgmap v432: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:36.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:36 smithi045 ceph-mon[81423]: pgmap v433: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:36.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:36 smithi044 ceph-mon[93020]: pgmap v433: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:38.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:38 smithi045 ceph-mon[81423]: pgmap v434: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:38.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:38 smithi044 ceph-mon[93020]: pgmap v434: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:40 smithi045 ceph-mon[81423]: pgmap v435: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:40.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:40 smithi044 ceph-mon[93020]: pgmap v435: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:41.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:41 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:41.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:41 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:42 smithi045 ceph-mon[81423]: pgmap v436: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:42 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:57:42.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:42 smithi044 ceph-mon[93020]: pgmap v436: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:42.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:42 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:57:43.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:43 smithi044 ceph-mon[93020]: pgmap v437: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:43.579 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:43 smithi045 ceph-mon[81423]: pgmap v437: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:44.377 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:44 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:57:44.377 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:44 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:57:44.377 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:44 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:57:44.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:44 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:57:44.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:44 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:57:44.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:44 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:57:45.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:45 smithi044 ceph-mon[93020]: pgmap v438: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:45.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:45 smithi045 ceph-mon[81423]: pgmap v438: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:47 smithi044 ceph-mon[93020]: pgmap v439: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:47 smithi045 ceph-mon[81423]: pgmap v439: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:50.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:49 smithi044 ceph-mon[93020]: pgmap v440: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:49 smithi045 ceph-mon[81423]: pgmap v440: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:51 smithi044 ceph-mon[93020]: pgmap v441: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:51 smithi045 ceph-mon[81423]: pgmap v441: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:53 smithi044 ceph-mon[93020]: pgmap v442: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:53 smithi045 ceph-mon[81423]: pgmap v442: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:56.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:55 smithi044 ceph-mon[93020]: pgmap v443: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:56.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:55 smithi045 ceph-mon[81423]: pgmap v443: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:57:58.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:57 smithi044 ceph-mon[93020]: pgmap v444: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:57:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:57 smithi045 ceph-mon[81423]: pgmap v444: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:57:59 smithi044 ceph-mon[93020]: pgmap v445: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:57:59 smithi045 ceph-mon[81423]: pgmap v445: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:01 smithi044 ceph-mon[93020]: pgmap v446: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:02.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:01 smithi045 ceph-mon[81423]: pgmap v446: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:03 smithi044 ceph-mon[93020]: pgmap v447: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:03 smithi045 ceph-mon[81423]: pgmap v447: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:06.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:05 smithi044 ceph-mon[93020]: pgmap v448: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:05 smithi045 ceph-mon[81423]: pgmap v448: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:08.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:07 smithi044 ceph-mon[93020]: pgmap v449: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:07 smithi045 ceph-mon[81423]: pgmap v449: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:10.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:09 smithi044 ceph-mon[93020]: pgmap v450: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:09 smithi045 ceph-mon[81423]: pgmap v450: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:11.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:10 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:11.231 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:10 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:12.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:11 smithi044 ceph-mon[93020]: pgmap v451: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:12.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:11 smithi045 ceph-mon[81423]: pgmap v451: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:14.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:13 smithi044 ceph-mon[93020]: pgmap v452: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:13 smithi045 ceph-mon[81423]: pgmap v452: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:16.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:15 smithi044 ceph-mon[93020]: pgmap v453: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:15 smithi045 ceph-mon[81423]: pgmap v453: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:18.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:17 smithi044 ceph-mon[93020]: pgmap v454: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:18.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:17 smithi045 ceph-mon[81423]: pgmap v454: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:20.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:19 smithi044 ceph-mon[93020]: pgmap v455: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:19 smithi045 ceph-mon[81423]: pgmap v455: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:22.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:21 smithi044 ceph-mon[93020]: pgmap v456: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:21 smithi045 ceph-mon[81423]: pgmap v456: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:24.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:23 smithi044 ceph-mon[93020]: pgmap v457: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:24.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:23 smithi045 ceph-mon[81423]: pgmap v457: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:26.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:25 smithi045 ceph-mon[81423]: pgmap v458: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:26.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:25 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:26.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:25 smithi044 ceph-mon[93020]: pgmap v458: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:26.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:25 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:28.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:27 smithi045 ceph-mon[81423]: pgmap v459: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:27 smithi044 ceph-mon[93020]: pgmap v459: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:30.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:29 smithi045 ceph-mon[81423]: pgmap v460: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:29 smithi044 ceph-mon[93020]: pgmap v460: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:31.726 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:31 smithi044 ceph-mon[93020]: pgmap v461: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:31.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:31 smithi045 ceph-mon[81423]: pgmap v461: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:34.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:33 smithi044 ceph-mon[93020]: pgmap v462: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:33 smithi045 ceph-mon[81423]: pgmap v462: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:36.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:35 smithi044 ceph-mon[93020]: pgmap v463: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:36.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:35 smithi045 ceph-mon[81423]: pgmap v463: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:38.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:37 smithi044 ceph-mon[93020]: pgmap v464: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:38.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:37 smithi045 ceph-mon[81423]: pgmap v464: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:40.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:39 smithi044 ceph-mon[93020]: pgmap v465: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:39 smithi045 ceph-mon[81423]: pgmap v465: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:41.232 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:41 smithi044 ceph-mon[93020]: pgmap v466: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:41 smithi045 ceph-mon[81423]: pgmap v466: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:44.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:43 smithi044 ceph-mon[93020]: pgmap v467: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:44.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:43 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:58:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:43 smithi045 ceph-mon[81423]: pgmap v467: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:43 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:58:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:45 smithi044 ceph-mon[93020]: pgmap v468: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:58:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:58:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:58:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:45 smithi045 ceph-mon[81423]: pgmap v468: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:58:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:58:46.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:58:48.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:47 smithi044 ceph-mon[93020]: pgmap v469: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:47 smithi045 ceph-mon[81423]: pgmap v469: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:50.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:49 smithi044 ceph-mon[93020]: pgmap v470: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:49 smithi045 ceph-mon[81423]: pgmap v470: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:52.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:51 smithi044 ceph-mon[93020]: pgmap v471: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:51 smithi045 ceph-mon[81423]: pgmap v471: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:54.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:53 smithi044 ceph-mon[93020]: pgmap v472: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:53 smithi045 ceph-mon[81423]: pgmap v472: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:56.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:55 smithi044 ceph-mon[93020]: pgmap v473: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:56.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:55 smithi045 ceph-mon[81423]: pgmap v473: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:58:58.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:57 smithi044 ceph-mon[93020]: pgmap v474: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:58:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:57 smithi045 ceph-mon[81423]: pgmap v474: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:00.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:58:59 smithi044 ceph-mon[93020]: pgmap v475: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:58:59 smithi045 ceph-mon[81423]: pgmap v475: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:02.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:01 smithi044 ceph-mon[93020]: pgmap v476: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:02.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:01 smithi045 ceph-mon[81423]: pgmap v476: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:04.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:03 smithi044 ceph-mon[93020]: pgmap v477: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:03 smithi045 ceph-mon[81423]: pgmap v477: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:05 smithi045 ceph-mon[81423]: pgmap v478: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:06.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:05 smithi044 ceph-mon[93020]: pgmap v478: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:07 smithi045 ceph-mon[81423]: pgmap v479: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:08.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:07 smithi044 ceph-mon[93020]: pgmap v479: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:09 smithi045 ceph-mon[81423]: pgmap v480: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:10.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:09 smithi044 ceph-mon[93020]: pgmap v480: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:12.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:12 smithi045 ceph-mon[81423]: pgmap v481: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:12.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:12 smithi044 ceph-mon[93020]: pgmap v481: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:14.278 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:14 smithi044 ceph-mon[93020]: pgmap v482: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:14 smithi045 ceph-mon[81423]: pgmap v482: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:16.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:16 smithi045 ceph-mon[81423]: pgmap v483: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:16.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:16 smithi044 ceph-mon[93020]: pgmap v483: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:18.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:18 smithi045 ceph-mon[81423]: pgmap v484: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:18 smithi044 ceph-mon[93020]: pgmap v484: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:20 smithi045 ceph-mon[81423]: pgmap v485: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:20 smithi044 ceph-mon[93020]: pgmap v485: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:22 smithi045 ceph-mon[81423]: pgmap v486: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:22.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:22 smithi044 ceph-mon[93020]: pgmap v486: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:24.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:24 smithi045 ceph-mon[81423]: pgmap v487: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:24.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:24 smithi044 ceph-mon[93020]: pgmap v487: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:26.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:26 smithi045 ceph-mon[81423]: pgmap v488: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:26.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:26.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:26 smithi044 ceph-mon[93020]: pgmap v488: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:26.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:28.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:28 smithi045 ceph-mon[81423]: pgmap v489: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:28.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:28 smithi044 ceph-mon[93020]: pgmap v489: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:30.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:30 smithi045 ceph-mon[81423]: pgmap v490: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:30.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:30 smithi044 ceph-mon[93020]: pgmap v490: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:32 smithi045 ceph-mon[81423]: pgmap v491: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:32.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:32 smithi044 ceph-mon[93020]: pgmap v491: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:34.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:34 smithi044 ceph-mon[93020]: pgmap v492: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:34.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:34 smithi045 ceph-mon[81423]: pgmap v492: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:35.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:35 smithi044 ceph-mon[93020]: pgmap v493: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:35.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:35 smithi045 ceph-mon[81423]: pgmap v493: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:38.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:37 smithi044 ceph-mon[93020]: pgmap v494: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:38.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:37 smithi045 ceph-mon[81423]: pgmap v494: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:39 smithi044 ceph-mon[93020]: pgmap v495: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:39 smithi045 ceph-mon[81423]: pgmap v495: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:41.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:41.232 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:42.226 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:41 smithi044 ceph-mon[93020]: pgmap v496: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:41 smithi045 ceph-mon[81423]: pgmap v496: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:44.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:43 smithi044 ceph-mon[93020]: pgmap v497: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:43 smithi045 ceph-mon[81423]: pgmap v497: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:45 smithi044 ceph-mon[93020]: pgmap v498: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:46.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:45 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:59:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:45 smithi045 ceph-mon[81423]: pgmap v498: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:45 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T11:59:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:59:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:59:47.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:46 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:59:47.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:46 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T11:59:47.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:46 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T11:59:47.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:46 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T11:59:48.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:47 smithi044 ceph-mon[93020]: pgmap v499: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:47 smithi045 ceph-mon[81423]: pgmap v499: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:50.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:49 smithi044 ceph-mon[93020]: pgmap v500: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:50.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:49 smithi045 ceph-mon[81423]: pgmap v500: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:52.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:51 smithi044 ceph-mon[93020]: pgmap v501: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:51 smithi045 ceph-mon[81423]: pgmap v501: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:54.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:53 smithi044 ceph-mon[93020]: pgmap v502: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:53 smithi045 ceph-mon[81423]: pgmap v502: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:56.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:55 smithi045 ceph-mon[81423]: pgmap v503: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:56.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:55 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:55 smithi044 ceph-mon[93020]: pgmap v503: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:55 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T11:59:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:57 smithi045 ceph-mon[81423]: pgmap v504: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T11:59:58.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:57 smithi044 ceph-mon[93020]: pgmap v504: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:00.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 11:59:59 smithi045 ceph-mon[81423]: pgmap v505: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:00.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 11:59:59 smithi044 ceph-mon[93020]: pgmap v505: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:01.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:01 smithi045 ceph-mon[81423]: overall HEALTH_OK 2024-04-16T12:00:01.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:01 smithi044 ceph-mon[93020]: overall HEALTH_OK 2024-04-16T12:00:02.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:02 smithi045 ceph-mon[81423]: pgmap v506: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:02.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:02 smithi044 ceph-mon[93020]: pgmap v506: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:04 smithi045 ceph-mon[81423]: pgmap v507: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:04.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:04 smithi044 ceph-mon[93020]: pgmap v507: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:06 smithi045 ceph-mon[81423]: pgmap v508: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:06.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:06 smithi044 ceph-mon[93020]: pgmap v508: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:08 smithi045 ceph-mon[81423]: pgmap v509: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:08.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:08 smithi044 ceph-mon[93020]: pgmap v509: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:10.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:10 smithi045 ceph-mon[81423]: pgmap v510: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:10.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:10 smithi044 ceph-mon[93020]: pgmap v510: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:11.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:11 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:11.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:11 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:12.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:12 smithi045 ceph-mon[81423]: pgmap v511: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:12.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:12 smithi044 ceph-mon[93020]: pgmap v511: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:14.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:14 smithi045 ceph-mon[81423]: pgmap v512: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:14.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:14 smithi044 ceph-mon[93020]: pgmap v512: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:16.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:16 smithi045 ceph-mon[81423]: pgmap v513: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:16.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:16 smithi044 ceph-mon[93020]: pgmap v513: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:18.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:18 smithi045 ceph-mon[81423]: pgmap v514: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:18.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:18 smithi044 ceph-mon[93020]: pgmap v514: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:20.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:20 smithi045 ceph-mon[81423]: pgmap v515: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:20.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:20 smithi044 ceph-mon[93020]: pgmap v515: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:22.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:22 smithi045 ceph-mon[81423]: pgmap v516: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:22.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:22 smithi044 ceph-mon[93020]: pgmap v516: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:24.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:24 smithi044 ceph-mon[93020]: pgmap v517: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:24.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:24 smithi045 ceph-mon[81423]: pgmap v517: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:26.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:26 smithi044 ceph-mon[93020]: pgmap v518: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:26.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:26 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:26.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:26 smithi045 ceph-mon[81423]: pgmap v518: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:26.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:26 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:27.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:27 smithi044 ceph-mon[93020]: pgmap v519: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:27.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:27 smithi045 ceph-mon[81423]: pgmap v519: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:30.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:29 smithi044 ceph-mon[93020]: pgmap v520: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:30.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:29 smithi045 ceph-mon[81423]: pgmap v520: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:32.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:31 smithi044 ceph-mon[93020]: pgmap v521: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:32.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:31 smithi045 ceph-mon[81423]: pgmap v521: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:34.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:33 smithi044 ceph-mon[93020]: pgmap v522: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:34.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:33 smithi045 ceph-mon[81423]: pgmap v522: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:36.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:35 smithi044 ceph-mon[93020]: pgmap v523: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:36.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:35 smithi045 ceph-mon[81423]: pgmap v523: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:38.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:37 smithi044 ceph-mon[93020]: pgmap v524: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:38.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:37 smithi045 ceph-mon[81423]: pgmap v524: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:40.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:39 smithi044 ceph-mon[93020]: pgmap v525: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:40.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:39 smithi045 ceph-mon[81423]: pgmap v525: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:41.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:40 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:41.232 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:40 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:42.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:41 smithi044 ceph-mon[93020]: pgmap v526: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:42.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:41 smithi045 ceph-mon[81423]: pgmap v526: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:44.227 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:43 smithi044 ceph-mon[93020]: pgmap v527: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:44.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:43 smithi045 ceph-mon[81423]: pgmap v527: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:46.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:45 smithi045 ceph-mon[81423]: pgmap v528: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:46.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:45 smithi044 ceph-mon[93020]: pgmap v528: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:48.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:47 smithi045 ceph-mon[81423]: pgmap v529: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:48.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:47 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T12:00:48.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:47 smithi044 ceph-mon[93020]: pgmap v529: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:48.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:47 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T12:00:49.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:48 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T12:00:49.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:48 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T12:00:49.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:48 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T12:00:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T12:00:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T12:00:49.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:48 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' 2024-04-16T12:00:50.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:49 smithi045 ceph-mon[81423]: pgmap v530: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:50.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:49 smithi044 ceph-mon[93020]: pgmap v530: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:52.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:52 smithi045 ceph-mon[81423]: pgmap v531: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:52.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:52 smithi044 ceph-mon[93020]: pgmap v531: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:54.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:54 smithi045 ceph-mon[81423]: pgmap v532: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:54.476 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:54 smithi044 ceph-mon[93020]: pgmap v532: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:56.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:56 smithi045 ceph-mon[81423]: pgmap v533: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:56.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:56 smithi045 ceph-mon[81423]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:56 smithi044 ceph-mon[93020]: pgmap v533: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:56.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:56 smithi044 ceph-mon[93020]: from='mgr.34104 172.21.15.44:0/2896935245' entity='mgr.smithi044.sczobv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T12:00:58.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:00:58 smithi045 ceph-mon[81423]: pgmap v534: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:00:58.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:00:58 smithi044 ceph-mon[93020]: pgmap v534: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:00.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:00 smithi045 ceph-mon[81423]: pgmap v535: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:00.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:00 smithi044 ceph-mon[93020]: pgmap v535: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:02.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:02 smithi045 ceph-mon[81423]: pgmap v536: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:02.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:02 smithi044 ceph-mon[93020]: pgmap v536: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:04.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:04 smithi045 ceph-mon[81423]: pgmap v537: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:04.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:04 smithi044 ceph-mon[93020]: pgmap v537: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:06.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:06 smithi045 ceph-mon[81423]: pgmap v538: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:06.477 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:06 smithi044 ceph-mon[93020]: pgmap v538: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:07.320 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-16T12:01:07.321 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T12:01:07.323 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-16T12:01:07.335 INFO:tasks.cephadm:Teardown begin 2024-04-16T12:01:07.335 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephadm.py", line 2236, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T12:01:07.336 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T12:01:07.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T12:01:07.560 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-16T12:01:07.560 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T12:01:07.620 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T12:01:07.679 INFO:tasks.cephadm:Stopping all daemons... 2024-04-16T12:01:07.680 INFO:tasks.cephadm.mon.smithi044:Stopping mon.smithi044... 2024-04-16T12:01:07.680 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044 2024-04-16T12:01:08.050 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:07 smithi044 systemd[1]: Stopping Ceph mon.smithi044 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:08.330 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:08 smithi044 ceph-mon[93020]: pgmap v539: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:08.330 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:08 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[93016]: 2024-04-16T12:01:08.295+0000 7f2290400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi044 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:08.330 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:08 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044[93016]: 2024-04-16T12:01:08.295+0000 7f2290400700 -1 mon.smithi044@0(leader) e3 *** Got Signal Terminated *** 2024-04-16T12:01:08.330 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:08 smithi045 ceph-mon[81423]: pgmap v539: 65 pgs: 65 active+clean; 584 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-16T12:01:08.683 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:08 smithi044 podman[149504]: 2024-04-16 12:01:08.368652195 +0000 UTC m=+0.235469093 container died 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-04-16T12:01:08.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:08 smithi044 podman[149504]: 2024-04-16 12:01:08.86959196 +0000 UTC m=+0.736408871 container cleanup 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-16T12:01:08.977 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:08 smithi044 bash[149504]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044 2024-04-16T12:01:09.727 INFO:journalctl@ceph.mon.smithi044.smithi044.stdout:Apr 16 12:01:09 smithi044 podman[149529]: 2024-04-16 12:01:09.237955379 +0000 UTC m=+0.855911807 container remove 5d963afffb61dc658593181d2cdd4bf9742b78832c581f1aab89aefba9fe17a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi044, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T12:01:09.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi044.service' 2024-04-16T12:01:09.977 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:09.978 INFO:tasks.cephadm.mon.smithi044:Stopped mon.smithi044 2024-04-16T12:01:09.978 INFO:tasks.cephadm.mon.smithi045:Stopping mon.smithi045... 2024-04-16T12:01:09.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi045 2024-04-16T12:01:10.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:10 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:10.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:10 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045[81418]: 2024-04-16T12:01:10.549+0000 7ffa33400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:10.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:10 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045[81418]: 2024-04-16T12:01:10.549+0000 7ffa33400700 -1 mon.smithi045@1(peon) e3 *** Got Signal Terminated *** 2024-04-16T12:01:10.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:10 smithi045 podman[118394]: 2024-04-16 12:01:10.715502505 +0000 UTC m=+0.283629696 container died 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0) 2024-04-16T12:01:11.250 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:11 smithi045 podman[118394]: 2024-04-16 12:01:11.226016627 +0000 UTC m=+0.794143808 container cleanup 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-16T12:01:11.250 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:11 smithi045 bash[118394]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045 2024-04-16T12:01:11.580 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Apr 16 12:01:11 smithi045 podman[118408]: 2024-04-16 12:01:11.49450397 +0000 UTC m=+0.763657397 container remove 72db4f2396ce18dfe1bcbf38f5853abb7bad57a04c26965af43f0186eff4079c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-mon-smithi045, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T12:01:11.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@mon.smithi045.service' 2024-04-16T12:01:12.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:12.150 INFO:tasks.cephadm.mon.smithi045:Stopped mon.smithi045 2024-04-16T12:01:12.150 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-16T12:01:12.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.0 2024-04-16T12:01:12.656 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:12 smithi044 systemd[1]: Stopping Ceph osd.0 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:12.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:12 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[102023]: 2024-04-16T12:01:12.715+0000 7f6b86c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:12.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:12 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[102023]: 2024-04-16T12:01:12.715+0000 7f6b86c00700 -1 osd.0 91 *** Got signal Terminated *** 2024-04-16T12:01:12.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:12 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0[102023]: 2024-04-16T12:01:12.715+0000 7f6b86c00700 -1 osd.0 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T12:01:18.597 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:18 smithi044 podman[149648]: 2024-04-16 12:01:18.273516654 +0000 UTC m=+5.722735807 container died 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-16T12:01:18.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:18 smithi044 podman[149648]: 2024-04-16 12:01:18.733191583 +0000 UTC m=+6.182410770 container cleanup 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T12:01:18.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:18 smithi044 bash[149648]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0 2024-04-16T12:01:19.477 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:19 smithi044 podman[149666]: 2024-04-16 12:01:19.001055752 +0000 UTC m=+0.716020818 container remove 93ed878736d9fafe5db3758c0773beae89c900d36f45cf704180184a59127ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-04-16T12:01:20.205 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:19 smithi044 podman[149754]: 2024-04-16 12:01:19.795242573 +0000 UTC m=+0.038202440 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T12:01:20.205 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:19 smithi044 podman[149754]: 2024-04-16 12:01:19.901982767 +0000 UTC m=+0.144942671 container create a4ef22a243326bbcb85b3da8f43cd9061d4fe43ae4a77a94d98c6e8c9b85ea18 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-16T12:01:20.205 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:20 smithi044 podman[149754]: 2024-04-16 12:01:20.206011469 +0000 UTC m=+0.448971400 container init a4ef22a243326bbcb85b3da8f43cd9061d4fe43ae4a77a94d98c6e8c9b85ea18 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-16T12:01:20.477 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:20 smithi044 podman[149754]: 2024-04-16 12:01:20.218562159 +0000 UTC m=+0.461522093 container start a4ef22a243326bbcb85b3da8f43cd9061d4fe43ae4a77a94d98c6e8c9b85ea18 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD) 2024-04-16T12:01:20.477 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:20 smithi044 podman[149754]: 2024-04-16 12:01:20.312965586 +0000 UTC m=+0.555925550 container attach a4ef22a243326bbcb85b3da8f43cd9061d4fe43ae4a77a94d98c6e8c9b85ea18 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T12:01:20.977 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 16 12:01:20 smithi044 podman[149754]: 2024-04-16 12:01:20.689936591 +0000 UTC m=+0.932896501 container died a4ef22a243326bbcb85b3da8f43cd9061d4fe43ae4a77a94d98c6e8c9b85ea18 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240409, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True) 2024-04-16T12:01:21.500 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-16T12:01:21.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:21.635 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-16T12:01:21.635 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-16T12:01:21.635 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.1 2024-04-16T12:01:22.227 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:21 smithi044 systemd[1]: Stopping Ceph osd.1 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:22.727 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:22 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[107672]: 2024-04-16T12:01:22.276+0000 7f69a1a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:22.727 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:22 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[107672]: 2024-04-16T12:01:22.276+0000 7f69a1a00700 -1 osd.1 91 *** Got signal Terminated *** 2024-04-16T12:01:22.727 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:22 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1[107672]: 2024-04-16T12:01:22.276+0000 7f69a1a00700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T12:01:28.126 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:27 smithi044 podman[149857]: 2024-04-16 12:01:27.789494304 +0000 UTC m=+5.635859238 container died 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T12:01:28.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:28 smithi044 podman[149857]: 2024-04-16 12:01:28.260960186 +0000 UTC m=+6.107325135 container cleanup 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T12:01:28.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:28 smithi044 bash[149857]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1 2024-04-16T12:01:28.977 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:28 smithi044 podman[149877]: 2024-04-16 12:01:28.507816754 +0000 UTC m=+0.703351038 container remove 93c9db3f124ef8fe92412c8da5deb6caa2d16a2b5a61ce49d98ecbae4a53245d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-16T12:01:29.694 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:29 smithi044 podman[149966]: 2024-04-16 12:01:29.293674205 +0000 UTC m=+0.037700647 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T12:01:29.694 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:29 smithi044 podman[149966]: 2024-04-16 12:01:29.430009402 +0000 UTC m=+0.174035874 container create 6ec85155322f6133a10dcff88184b9a10d6698a31e6b95d5a5ed81ca817d77d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-16T12:01:29.694 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:29 smithi044 podman[149966]: 2024-04-16 12:01:29.695084688 +0000 UTC m=+0.439111166 container init 6ec85155322f6133a10dcff88184b9a10d6698a31e6b95d5a5ed81ca817d77d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-16T12:01:29.977 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:29 smithi044 podman[149966]: 2024-04-16 12:01:29.707393458 +0000 UTC m=+0.451419903 container start 6ec85155322f6133a10dcff88184b9a10d6698a31e6b95d5a5ed81ca817d77d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T12:01:29.977 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:29 smithi044 podman[149966]: 2024-04-16 12:01:29.841562 +0000 UTC m=+0.585588492 container attach 6ec85155322f6133a10dcff88184b9a10d6698a31e6b95d5a5ed81ca817d77d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T12:01:30.477 INFO:journalctl@ceph.osd.1.smithi044.stdout:Apr 16 12:01:30 smithi044 podman[149966]: 2024-04-16 12:01:30.18101057 +0000 UTC m=+0.925037006 container died 6ec85155322f6133a10dcff88184b9a10d6698a31e6b95d5a5ed81ca817d77d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-16T12:01:31.009 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-16T12:01:31.130 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:31.131 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-16T12:01:31.131 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-16T12:01:31.131 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2 2024-04-16T12:01:31.727 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:31 smithi044 systemd[1]: Stopping Ceph osd.2 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:32.227 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:31 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[113023]: 2024-04-16T12:01:31.806+0000 7fef38c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:32.227 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:31 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[113023]: 2024-04-16T12:01:31.806+0000 7fef38c00700 -1 osd.2 91 *** Got signal Terminated *** 2024-04-16T12:01:32.227 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:31 smithi044 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2[113023]: 2024-04-16T12:01:31.806+0000 7fef38c00700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T12:01:37.600 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:37 smithi044 podman[150070]: 2024-04-16 12:01:37.275518857 +0000 UTC m=+5.638536485 container died b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-16T12:01:37.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:37 smithi044 podman[150070]: 2024-04-16 12:01:37.738621651 +0000 UTC m=+6.101639303 container cleanup b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-16T12:01:37.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:37 smithi044 bash[150070]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2 2024-04-16T12:01:38.477 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:37 smithi044 podman[150088]: 2024-04-16 12:01:37.997841094 +0000 UTC m=+0.706611932 container remove b3bcfc1b4f971962fe4796c0514cb7fc2abae3606f474e297bbc639be0fec4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-16T12:01:39.193 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:38 smithi044 podman[150177]: 2024-04-16 12:01:38.793031831 +0000 UTC m=+0.034438720 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T12:01:39.193 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:38 smithi044 podman[150177]: 2024-04-16 12:01:38.890487259 +0000 UTC m=+0.131894190 container create e109c395008b340c7ab265276be8d35b8b6586827b4874e3fcc54318272d3b26 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409) 2024-04-16T12:01:39.193 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:39 smithi044 podman[150177]: 2024-04-16 12:01:39.194049408 +0000 UTC m=+0.435456325 container init e109c395008b340c7ab265276be8d35b8b6586827b4874e3fcc54318272d3b26 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2) 2024-04-16T12:01:39.477 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:39 smithi044 podman[150177]: 2024-04-16 12:01:39.207544125 +0000 UTC m=+0.448951018 container start e109c395008b340c7ab265276be8d35b8b6586827b4874e3fcc54318272d3b26 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T12:01:39.477 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:39 smithi044 podman[150177]: 2024-04-16 12:01:39.292702381 +0000 UTC m=+0.534109298 container attach e109c395008b340c7ab265276be8d35b8b6586827b4874e3fcc54318272d3b26 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T12:01:39.977 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:39 smithi044 podman[150177]: 2024-04-16 12:01:39.674149704 +0000 UTC m=+0.915556642 container died e109c395008b340c7ab265276be8d35b8b6586827b4874e3fcc54318272d3b26 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T12:01:40.434 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:40 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:40.173+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:40.707 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-16T12:01:40.802 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:40 smithi044 podman[150202]: 2024-04-16 12:01:40.663106255 +0000 UTC m=+0.975435755 container remove e109c395008b340c7ab265276be8d35b8b6586827b4874e3fcc54318272d3b26 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T12:01:40.802 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:40 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-16T12:01:40.802 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:40 smithi044 systemd[1]: Stopped Ceph osd.2 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T12:01:40.802 INFO:journalctl@ceph.osd.2.smithi044.stdout:Apr 16 12:01:40 smithi044 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.2.service: Consumed 16.052s CPU time. 2024-04-16T12:01:40.830 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:40 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:40.435+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:40.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:40.846 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-16T12:01:40.846 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-16T12:01:40.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3 2024-04-16T12:01:41.196 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:40 smithi045 systemd[1]: Stopping Ceph osd.3 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:41.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:41 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:41.390+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:41.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:41 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:41.197+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:41.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:41 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:41.435+0000 7f067f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:41.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:41 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:41.435+0000 7f067f200700 -1 osd.3 91 *** Got signal Terminated *** 2024-04-16T12:01:41.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:41 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:41.435+0000 7f067f200700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T12:01:42.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:41 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:41.700+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:42.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:42 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:42.163+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:42.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:42 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:42.400+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:43.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:42 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:42.708+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:43.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:43.430+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:43.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:43.181+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:44.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:43 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:43.701+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:44.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:44 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:44.459+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:44.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:44 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:44.216+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:45.080 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:44 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:44.684+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:45.461 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:45 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:45.201+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:45.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:45 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:45.668+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:45.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:45 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:45.461+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:46.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:46 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3[90736]: 2024-04-16T12:01:46.231+0000 7f067aa00700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:14.317662+0000 front 2024-04-16T12:01:14.317759+0000 (oldest deadline 2024-04-16T12:01:39.617143+0000) 2024-04-16T12:01:46.582 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:46 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:46.444+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:47.080 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:46 smithi045 podman[118515]: 2024-04-16 12:01:46.894935533 +0000 UTC m=+5.584763096 container died be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T12:01:47.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:46 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:46.698+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:47.503 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:47 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:47.427+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:47.503 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:47 smithi045 podman[118515]: 2024-04-16 12:01:47.303062147 +0000 UTC m=+5.992889703 container cleanup be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T12:01:47.503 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:47 smithi045 bash[118515]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3 2024-04-16T12:01:47.831 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:47 smithi045 podman[118533]: 2024-04-16 12:01:47.504223814 +0000 UTC m=+0.588356261 container remove be3375e68a6a44cd9b38747f15e625c26e23a4a6521e089c5479e1ae343bcf40 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T12:01:47.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:47 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:47.689+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:48.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:48 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:48.426+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:48.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:48 smithi045 podman[118623]: 2024-04-16 12:01:48.220866332 +0000 UTC m=+0.035072331 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T12:01:48.581 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:48 smithi045 podman[118623]: 2024-04-16 12:01:48.330258815 +0000 UTC m=+0.144464847 container create 4059478d5c2116329f94143ffa35fc9a83fd084a58f56564ca12f11c705996af (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T12:01:49.077 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:48 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:48.689+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:49.077 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:48 smithi045 podman[118623]: 2024-04-16 12:01:48.592840318 +0000 UTC m=+0.407046407 container init 4059478d5c2116329f94143ffa35fc9a83fd084a58f56564ca12f11c705996af (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T12:01:49.077 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:48 smithi045 podman[118623]: 2024-04-16 12:01:48.605649539 +0000 UTC m=+0.419855591 container start 4059478d5c2116329f94143ffa35fc9a83fd084a58f56564ca12f11c705996af (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True) 2024-04-16T12:01:49.077 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:48 smithi045 podman[118623]: 2024-04-16 12:01:48.68247145 +0000 UTC m=+0.496677502 container attach 4059478d5c2116329f94143ffa35fc9a83fd084a58f56564ca12f11c705996af (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T12:01:49.077 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:49 smithi045 podman[118623]: 2024-04-16 12:01:49.078213166 +0000 UTC m=+0.892419252 container died 4059478d5c2116329f94143ffa35fc9a83fd084a58f56564ca12f11c705996af (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-16T12:01:49.690 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:49.471+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:49.690 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:49.472+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:49.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-16T12:01:50.027 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:49.691+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:50.027 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:49 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:49.691+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:50.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:49 smithi045 podman[118648]: 2024-04-16 12:01:49.902094701 +0000 UTC m=+0.809084911 container remove 4059478d5c2116329f94143ffa35fc9a83fd084a58f56564ca12f11c705996af (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T12:01:50.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:49 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-16T12:01:50.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:49 smithi045 systemd[1]: Stopped Ceph osd.3 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968. 2024-04-16T12:01:50.027 INFO:journalctl@ceph.osd.3.smithi045.stdout:Apr 16 12:01:49 smithi045 systemd[1]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.3.service: Consumed 15.793s CPU time. 2024-04-16T12:01:50.069 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:50.069 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-16T12:01:50.069 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-16T12:01:50.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.4 2024-04-16T12:01:50.331 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:50 smithi045 systemd[1]: Stopping Ceph osd.4 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:01:50.740 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:50.484+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:50.740 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:50.484+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:51.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:50.740+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:51.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:50.740+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:51.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:50.754+0000 7f2d33a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:01:51.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:50.754+0000 7f2d33a00700 -1 osd.4 91 *** Got signal Terminated *** 2024-04-16T12:01:51.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:50 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:50.754+0000 7f2d33a00700 -1 osd.4 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T12:01:51.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:51 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:51.725+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:51.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:51 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:51.725+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:51.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:51 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:51.532+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:51.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:51 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:51.532+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:52.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:52 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:52.730+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:52.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:52 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:52.730+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:52.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:52 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:52.505+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:52.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:52 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:52.505+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:53.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:53 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:53.694+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:53.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:53 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:53.694+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:53.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:53 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:53.484+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:53.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:53 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:53.484+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:54.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:54 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:54.522+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:54.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:54 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:54.522+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:54.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:54 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:54.684+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:54.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:54 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:54.684+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:55.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:55 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:55.687+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:15.087239+0000 front 2024-04-16T12:01:15.087013+0000 (oldest deadline 2024-04-16T12:01:40.986828+0000) 2024-04-16T12:01:55.831 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:55 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4[96028]: 2024-04-16T12:01:55.687+0000 7f2d2f200700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:24.389652+0000 front 2024-04-16T12:01:24.389466+0000 (oldest deadline 2024-04-16T12:01:49.089030+0000) 2024-04-16T12:01:55.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:55 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:55.484+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:55.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:55 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:55.484+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:56.581 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:56 smithi045 podman[118729]: 2024-04-16 12:01:56.276401735 +0000 UTC m=+5.684714803 container died c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T12:01:56.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:56 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:56.521+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:56.581 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:56 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:56.521+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:57.054 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:56 smithi045 podman[118729]: 2024-04-16 12:01:56.779774976 +0000 UTC m=+6.188088059 container cleanup c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T12:01:57.055 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:56 smithi045 bash[118729]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4 2024-04-16T12:01:57.331 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:57 smithi045 podman[118747]: 2024-04-16 12:01:57.055690679 +0000 UTC m=+0.763332137 container remove c6a09eb63df3d2e0b8f844cdf69e1073023daa27bb2d00577bb0fbeca73c642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-16T12:01:57.830 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:57 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:57.547+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:57.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:57 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:57.547+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:58.225 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:57 smithi045 podman[118832]: 2024-04-16 12:01:57.828028593 +0000 UTC m=+0.035543997 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T12:01:58.225 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:58 smithi045 podman[118832]: 2024-04-16 12:01:58.056474279 +0000 UTC m=+0.263989714 container create dd81779dd2a2c0d219ff6a00c0fb6f61329fb40ecb1c76a6224f02ba79cd9a58 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5) 2024-04-16T12:01:58.535 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:58 smithi045 podman[118832]: 2024-04-16 12:01:58.329165298 +0000 UTC m=+0.536680742 container init dd81779dd2a2c0d219ff6a00c0fb6f61329fb40ecb1c76a6224f02ba79cd9a58 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T12:01:58.535 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:58 smithi045 podman[118832]: 2024-04-16 12:01:58.341774158 +0000 UTC m=+0.549289636 container start dd81779dd2a2c0d219ff6a00c0fb6f61329fb40ecb1c76a6224f02ba79cd9a58 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-16T12:01:58.535 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:58 smithi045 podman[118832]: 2024-04-16 12:01:58.417078513 +0000 UTC m=+0.624593988 container attach dd81779dd2a2c0d219ff6a00c0fb6f61329fb40ecb1c76a6224f02ba79cd9a58 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-16T12:01:58.808 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:58 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:58.536+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:58.808 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:58 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:58.536+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:59.081 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 16 12:01:58 smithi045 podman[118832]: 2024-04-16 12:01:58.810565617 +0000 UTC m=+1.018081111 container died dd81779dd2a2c0d219ff6a00c0fb6f61329fb40ecb1c76a6224f02ba79cd9a58 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-16T12:01:59.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-16T12:01:59.787 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:59 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:59.531+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:01:59.788 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:59 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:59.531+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:01:59.788 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:59 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:01:59.531+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-04-16T12:01:34.438907+0000 front 2024-04-16T12:01:34.439071+0000 (oldest deadline 2024-04-16T12:01:59.138435+0000) 2024-04-16T12:01:59.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:01:59.792 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-16T12:01:59.792 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-16T12:01:59.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.5 2024-04-16T12:02:00.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:01:59 smithi045 systemd[1]: Stopping Ceph osd.5 for 8b0307a8-fbe4-11ee-bc8f-c7b262605968... 2024-04-16T12:02:00.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:00 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:00.374+0000 7f2f2b200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T12:02:00.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:00 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:00.374+0000 7f2f2b200700 -1 osd.5 91 *** Got signal Terminated *** 2024-04-16T12:02:00.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:00 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:00.374+0000 7f2f2b200700 -1 osd.5 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T12:02:00.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:00 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:00.577+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:02:00.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:00 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:00.577+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:02:00.832 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:00 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:00.577+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-04-16T12:01:34.438907+0000 front 2024-04-16T12:01:34.439071+0000 (oldest deadline 2024-04-16T12:01:59.138435+0000) 2024-04-16T12:02:02.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:01 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:01.612+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:02:02.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:01 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:01.612+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:02:02.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:01 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:01.612+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-04-16T12:01:34.438907+0000 front 2024-04-16T12:01:34.439071+0000 (oldest deadline 2024-04-16T12:01:59.138435+0000) 2024-04-16T12:02:03.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:02 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:02.655+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:02:03.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:02 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:02.655+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:02:03.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:02 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:02.655+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-04-16T12:01:34.438907+0000 front 2024-04-16T12:01:34.439071+0000 (oldest deadline 2024-04-16T12:01:59.138435+0000) 2024-04-16T12:02:04.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:03 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:03.665+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:02:04.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:03 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:03.665+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:02:04.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:03 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:03.665+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-04-16T12:01:34.438907+0000 front 2024-04-16T12:01:34.439071+0000 (oldest deadline 2024-04-16T12:01:59.138435+0000) 2024-04-16T12:02:05.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:04 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:04.689+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-16T12:01:17.034684+0000 front 2024-04-16T12:01:17.034954+0000 (oldest deadline 2024-04-16T12:01:39.934884+0000) 2024-04-16T12:02:05.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:04 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:04.689+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-04-16T12:01:23.436673+0000 front 2024-04-16T12:01:23.436499+0000 (oldest deadline 2024-04-16T12:01:48.736079+0000) 2024-04-16T12:02:05.081 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:04 smithi045 ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5[101334]: 2024-04-16T12:02:04.689+0000 7f2f21a00700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-04-16T12:01:34.438907+0000 front 2024-04-16T12:01:34.439071+0000 (oldest deadline 2024-04-16T12:01:59.138435+0000) 2024-04-16T12:02:06.137 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:05 smithi045 podman[118938]: 2024-04-16 12:02:05.8597305 +0000 UTC m=+5.611130030 container died 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T12:02:06.541 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:06 smithi045 podman[118938]: 2024-04-16 12:02:06.249543401 +0000 UTC m=+6.000942908 container cleanup 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T12:02:06.541 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:06 smithi045 bash[118938]: ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5 2024-04-16T12:02:06.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:06 smithi045 podman[118958]: 2024-04-16 12:02:06.542142847 +0000 UTC m=+0.671847031 container remove 114e11d6d49a38d5ca94673b8c07ed2e394ba1f7622b33417ef361d9cbc1c60a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T12:02:07.383 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:07 smithi045 podman[119047]: 2024-04-16 12:02:07.023734227 +0000 UTC m=+0.034580598 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T12:02:07.383 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:07 smithi045 podman[119047]: 2024-04-16 12:02:07.251179256 +0000 UTC m=+0.262025661 container create b40fc40ccb93ec51050c216b893ca8c4a691307a6a2bf8012fd7dbe461e50415 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T12:02:07.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:07 smithi045 podman[119047]: 2024-04-16 12:02:07.481023718 +0000 UTC m=+0.491870156 container init b40fc40ccb93ec51050c216b893ca8c4a691307a6a2bf8012fd7dbe461e50415 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T12:02:07.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:07 smithi045 podman[119047]: 2024-04-16 12:02:07.49352391 +0000 UTC m=+0.504370275 container start b40fc40ccb93ec51050c216b893ca8c4a691307a6a2bf8012fd7dbe461e50415 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T12:02:07.831 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:07 smithi045 podman[119047]: 2024-04-16 12:02:07.561571259 +0000 UTC m=+0.572417668 container attach b40fc40ccb93ec51050c216b893ca8c4a691307a6a2bf8012fd7dbe461e50415 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-16T12:02:08.275 INFO:journalctl@ceph.osd.5.smithi045.stdout:Apr 16 12:02:08 smithi045 podman[119073]: 2024-04-16 12:02:08.009367742 +0000 UTC m=+0.038585294 container died b40fc40ccb93ec51050c216b893ca8c4a691307a6a2bf8012fd7dbe461e50415 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968-osd-5-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T12:02:08.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b0307a8-fbe4-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-16T12:02:08.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T12:02:08.757 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-16T12:02:08.758 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T12:02:14.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stdout:ceph_fuse: fuse loop returned - joining 2024-04-16T12:02:14.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stdout:ceph_fuse: fuse loop returned - joined 2024-04-16T12:02:14.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:2024-04-16T12:02:14.608+0000 7fada952f300 -1 fuse_session_loop_mt 3 -- done 2024-04-16T12:02:14.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stdout:calling unmount 2024-04-16T12:02:14.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stderr:ceph-fuse[76454]: fuse finished with error 0 and tester_r 0 2024-04-16T12:02:39.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T12:02:44.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi044.stdout:calling finalize 2024-04-16T12:02:58.289 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T12:02:58.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T12:02:58.509 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-16T12:02:58.511 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/remote/smithi044/crash 2024-04-16T12:02:58.512 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T12:02:58.633 INFO:teuthology.orchestra.run.smithi044.stderr:tar: /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T12:02:58.633 INFO:teuthology.orchestra.run.smithi044.stderr:tar: Error is not recoverable: exiting now 2024-04-16T12:02:58.639 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/remote/smithi045/crash 2024-04-16T12:02:58.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T12:02:58.748 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /var/lib/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T12:02:58.749 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now 2024-04-16T12:02:58.752 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-16T12:02:58.752 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-16T12:02:58.871 INFO:tasks.cephadm:Compressing logs... 2024-04-16T12:02:58.871 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T12:02:58.875 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T12:02:58.940 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-16T12:02:58.944 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T12:02:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-16T12:02:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi044.ufhhlc.log 2024-04-16T12:02:58.947 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi044.hlafhv.log 2024-04-16T12:02:58.948 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi044.ufhhlc.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T12:02:58.949 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi044.hlafhv.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-16T12:02:58.951 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mgr.smithi044.sczobv.log 2024-04-16T12:02:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-16T12:02:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mgr.smithi044.sczobv.log: /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mon.smithi044.log 2024-04-16T12:02:58.957 INFO:teuthology.orchestra.run.smithi044.stderr: 91.1% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T12:02:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mon.smithi044.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T12:02:58.961 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T12:02:58.964 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip 93.5% -5 --verbose -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi044.hlafhv.log.gz -- 2024-04-16T12:02:58.964 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-16T12:02:58.965 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log: --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T12:02:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.2.log: 85.4% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T12:02:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T12:02:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:: No such file or directory 2024-04-16T12:02:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T12:02:58.974 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi045.hxlkks.log 2024-04-16T12:02:58.975 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-volume.log: 96.5% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-16T12:02:58.975 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-16T12:02:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi045.hxlkks.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-16T12:02:58.977 INFO:teuthology.orchestra.run.smithi045.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T12:02:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mon.smithi045.log 2024-04-16T12:02:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-16T12:02:58.981 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mon.smithi045.log: 91.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T12:02:58.982 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mgr.smithi045.sdlwif.log 2024-04-16T12:02:58.982 INFO:teuthology.orchestra.run.smithi044.stderr: 88.8% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T12:02:58.983 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi045.noofsd.log 2024-04-16T12:02:58.984 INFO:teuthology.orchestra.run.smithi044.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T12:02:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mgr.smithi045.sdlwif.log: gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T12:02:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi045.noofsd.log: /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.cephadm.log: 85.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T12:02:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T12:02:58.989 INFO:teuthology.orchestra.run.smithi045.stderr: 93.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi045.hxlkks.log.gz 2024-04-16T12:02:58.990 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T12:02:58.992 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-16T12:02:59.000 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-volume.log: /var/log/ceph/ceph-client.1.log: /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log: 88.9% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T12:02:59.004 INFO:teuthology.orchestra.run.smithi045.stderr: 93.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi045.noofsd.log.gz 2024-04-16T12:02:59.004 INFO:teuthology.orchestra.run.smithi044.stderr: 90.8% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T12:02:59.019 INFO:teuthology.orchestra.run.smithi045.stderr: 90.9% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T12:02:59.024 INFO:teuthology.orchestra.run.smithi045.stderr: 96.6% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-16T12:02:59.047 INFO:teuthology.orchestra.run.smithi045.stderr: 89.8% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mgr.smithi045.sdlwif.log.gz 2024-04-16T12:02:59.143 INFO:teuthology.orchestra.run.smithi044.stderr: 94.0% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi044.ufhhlc.log.gz 2024-04-16T12:02:59.338 INFO:teuthology.orchestra.run.smithi044.stderr: 88.5% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mgr.smithi044.sczobv.log.gz 2024-04-16T12:02:59.339 INFO:teuthology.orchestra.run.smithi045.stderr: 92.5% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mon.smithi045.log.gz 2024-04-16T12:02:59.616 INFO:teuthology.orchestra.run.smithi045.stderr: 93.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-16T12:02:59.653 INFO:teuthology.orchestra.run.smithi044.stderr: 93.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-16T12:02:59.663 INFO:teuthology.orchestra.run.smithi044.stderr: 93.3% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-16T12:02:59.697 INFO:teuthology.orchestra.run.smithi045.stderr: 93.7% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-16T12:02:59.783 INFO:teuthology.orchestra.run.smithi045.stderr: 93.4% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-16T12:02:59.787 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-16T12:02:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m0.856s 2024-04-16T12:02:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m2.475s 2024-04-16T12:02:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.320s 2024-04-16T12:02:59.844 INFO:teuthology.orchestra.run.smithi044.stderr: 93.4% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-16T12:03:00.063 INFO:teuthology.orchestra.run.smithi044.stderr: 90.4% -- replaced with /var/log/ceph/8b0307a8-fbe4-11ee-bc8f-c7b262605968/ceph-mon.smithi044.log.gz 2024-04-16T12:03:00.067 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-16T12:03:00.067 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m1.159s 2024-04-16T12:03:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:user 0m3.774s 2024-04-16T12:03:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m0.406s 2024-04-16T12:03:00.068 INFO:tasks.cephadm:Archiving logs... 2024-04-16T12:03:00.069 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/remote/smithi044/log 2024-04-16T12:03:00.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T12:03:00.667 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/remote/smithi045/log 2024-04-16T12:03:00.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T12:03:01.110 INFO:tasks.cephadm:Removing cluster... 2024-04-16T12:03:01.110 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 --force 2024-04-16T12:03:02.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b0307a8-fbe4-11ee-bc8f-c7b262605968 --force 2024-04-16T12:03:03.257 INFO:tasks.cephadm:Removing cephadm ... 2024-04-16T12:03:03.258 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T12:03:03.318 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T12:03:03.378 INFO:tasks.cephadm:Teardown complete 2024-04-16T12:03:03.378 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T12:03:03.392 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T12:03:03.393 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T12:03:03.393 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T12:03:03.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T12:03:03.557 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T12:03:03.569 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T12:03:03.569 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T12:03:03.577 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T12:03:03.626 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-04-16T12:03:03.627 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-04-16T12:03:03.637 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T12:03:03.637 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-04-16T12:03:03.637 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 +463us[ +486us] +/- 49ms 2024-04-16T12:03:03.637 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 -1024us[-1001us] +/- 59ms 2024-04-16T12:03:03.637 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +518us[ +541us] +/- 68ms 2024-04-16T12:03:03.637 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T12:03:03.638 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T12:03:03.638 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-04-16T12:03:03.638 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +462us[ +463us] +/- 49ms 2024-04-16T12:03:03.638 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 -1029us[-1027us] +/- 59ms 2024-04-16T12:03:03.638 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +490us[ +491us] +/- 68ms 2024-04-16T12:03:03.638 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T12:03:03.640 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T12:03:03.651 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T12:03:03.652 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T12:03:03.666 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T12:03:03.925 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T12:03:04.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T12:03:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T12:03:04.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T12:03:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T12:03:04.380 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T12:03:04.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T12:03:04.410 INFO:teuthology.task.internal:Duration was 2584.093137 seconds 2024-04-16T12:03:04.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T12:03:04.428 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T12:03:04.428 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T12:03:04.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T12:03:04.594 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T12:03:04.619 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T12:03:05.082 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T12:03:05.082 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-04-16T12:03:05.083 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T12:03:05.149 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-04-16T12:03:05.150 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T12:03:05.230 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T12:03:05.230 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T12:03:05.239 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T12:03:05.337 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T12:03:05.337 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T12:03:05.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T12:03:06.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T12:03:06.147 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T12:03:06.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T12:03:06.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T12:03:06.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T12:03:06.368 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T12:03:06.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T12:03:06.483 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-04-16T12:03:06.487 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-04-16T12:03:06.554 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T12:03:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T12:03:06.610 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T12:03:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T12:03:06.668 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T12:03:06.681 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T12:03:06.682 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/remote/smithi044 2024-04-16T12:03:06.682 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T12:03:06.814 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071/remote/smithi045 2024-04-16T12:03:06.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T12:03:06.931 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T12:03:06.931 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T12:03:06.940 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T12:03:07.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T12:03:07.014 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T12:03:07.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T12:03:07.028 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T12:03:07.028 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T12:03:07.035 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T12:03:07.088 INFO:teuthology.orchestra.run.smithi044.stdout: 393339 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 12:03 /home/ubuntu/cephtest 2024-04-16T12:03:07.089 INFO:teuthology.orchestra.run.smithi044.stdout: 799145 4 d--------- 2 ubuntu ubuntu 4096 Apr 16 11:40 /home/ubuntu/cephtest/mnt.0 2024-04-16T12:03:07.089 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-16T12:03:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T12:03:07.092 INFO:teuthology.orchestra.run.smithi044.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T12:03:07.092 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T12:03:07.093 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T12:03:07.107 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T12:03:07.120 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T12:03:07.120 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T12:03:07.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi045.stderr:unique: 706, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120528 2024-04-16T12:03:07.147 DEBUG:teuthology.nuke:shortname: smithi044 2024-04-16T12:03:07.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T12:03:07.171 DEBUG:teuthology.nuke:shortname: smithi045 2024-04-16T12:03:07.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T12:03:07.175 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 11:11:05.089780', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6Bk5baccvIJJs+tGtUEIPlBdRsUeJyJlsHTonddFEJdBksjvfLxzn/x1+9a4XK5hmD63IZlRVFjCAPB+kVKNw='} 2024-04-16T12:03:07.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_11:10:55-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7658071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 11:11:05.087295', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYlwGYIuSCqXHAUcISzjQxeGmJRcvFIiDt2jY7gf2HM4FuIi03ghhWtNQbELudwXDb7dYy73Vzte/GTo/A6X3I='} 2024-04-16T12:03:07.224 INFO:teuthology.orchestra.console.smithi044:Power off 2024-04-16T12:03:07.224 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T12:03:07.262 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-16T12:03:07.263 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T12:03:07.278 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2024-04-16T12:03:07.279 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:07.288 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-16T12:03:07.288 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:07.301 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-04-16T12:03:07.309 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-16T12:03:11.302 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:11.309 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:11.412 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-04-16T12:03:11.418 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-16T12:03:15.417 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:15.427 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:15.530 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2024-04-16T12:03:15.537 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-16T12:03:15.631 INFO:teuthology.orchestra.console.smithi044:Power off completed 2024-04-16T12:03:19.541 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T12:03:19.653 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-16T12:03:19.753 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-16T12:03:19.858 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2584.0931372642517 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 host.b-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 owner: scheduled_vshankar@teuthology status: fail success: false 2024-04-16T12:03:19.859 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T12:03:19.953 INFO:teuthology.run:FAIL